Superconducting qubits are an attractive platform for quantum computing since they have demonstrated high-fidelity quantum gates and extensibility to modest system sizes. Nonetheless, an outstanding challenge is stabilizing their energy-relaxation times, which can fluctuate unpredictably in frequency and time. Here, we use qubits as spectral and temporal probes of individual two-level-system defects to provide direct evidence that they are responsible for the largest fluctuations. This research lays the foundation for stabilizing qubit performance through calibration, design, and fabrication.