Hearing impairment represents the most common sensory deficit in humans. Genetic mutations contribute significantly to this disorder. Mostly, only malfunction of the ear is considered. Here, we assessed the role of the peripheral deafness gene Cacna1d, encoding the L-type channel Ca(v)1.3, in downstream processing of acoustic information. To this end, we generated a mouse conditional Cacna1d-eGFP(flex) allele. Upon pairing with Egr2::Cre mice, Ca(v)1.3 was ablated in the auditory brainstem, leaving the inner ear intact. Structural assessment of the superior olivary complex (SOC), an essential auditory brainstem center, revealed a dramatic volume reduction (43-47%) of major nuclei in young adult Egr2::Cre;Cacna1d-eGFP(flex) mice. This volume decline was mainly caused by a reduced cell number (decline by 46-56%). Abnormal formation of the lateral superior olive was already present at P4, demonstrating an essential perinatal role of Ca(v)1.3 in the SOC. Measurements of auditory brainstem responses demonstrated a decreased amplitude in the auditory nerve between 50 and 75 dB stimulation in Egr2::Cre;Cacna1d-eGFP(flex) knockout mice and increased amplitudes in central auditory processing centers. Immunohistochemical studies linked the amplitude changes in the central auditory system to reduced expression of K(v)1.2. No changes were observed for K(v)1.1, KCC2, a determinant of inhibitory neurotransmission, and choline acetyltransferase, a marker of efferent olivocochlear neurons. Together, these analyses identify a crucial retrocochlear role of Ca(v)1.3 and demonstrate that mutations in deafness genes can affect sensory cells and neurons alike. As a corollary, hearing aids have to address central auditory processing deficits as well.