The rhombic lip (RL) is the neuroepithelium immediately adjacent to the roof plate of the fourth ventricle, and it gives rise to various brainstem and cerebellar cell types. Our study shows that the bHLH (basic helix-loop-helix) transcription factor Olig3 is expressed in the progenitors of RL, and ablation of Olig3 significantly affects the development of RL. In Olig3-/- caudal RL, the expression level of Math1 in the dorsal interneuron 1 (dI1) domain is reduced, and the formation of four mossy-fiber nuclei is compromised; dI2-dI3 neurons are misspecified to dI4 interneurons, and the climbing-fiber neurons (inferior olive nucleus) are completely lost. In addition, the formation of brainstem (nor)adrenergic centers and first-order relay visceral sensory neurons is also dependent on Olig3. Therefore, Olig3 plays an important role in the fate specification and differentiation of caudal RL-derived neurons.