We discuss a fundamental limitation for the coherent operation of superconducting quantum bits originating from phonon radiation generated in the Josephson junctions of the device. The time dependent superconducting phase across the junction produces an electric field that couples to the underlying crystal lattice via the piezoelectric effect. We determine the radiation resistance of the junction due to phonon emission and derive substantial decoherence rates for the quantum bits, which are compatible with quality factors measured in recent experiments.