Although Josephson junction qubits show great promise for quantum computing, the origin of dominant decoherence mechanisms remains unknown. Improving the operation of a Josephson junction based phase qubit has revealed microscopic two-level systems or resonators within the tunnel barrier that cause decoherence. We report spectroscopic data that show a level splitting characteristic of coupling between a two-state qubit and a two-level system. Furthermore, we show Rabi oscillations whose "coherence amplitude" is significantly degraded by the presence of these spurious microwave resonators. The discovery of these resonators impacts the future of Josephson qubits as well as existing Josephson technologies.