How neural circuits associated with sexually dimorphic organs are differentially assembled during development is unclear. Here, we report a sexually dimorphic pattern of mouse mammary gland sensory innervation and the mechanism of its formation. Brain-derived neurotrophic factor (BDNF), emanating from mammary mesenchyme and signaling through its receptor TrkB on sensory axons, is required for establishing mammary gland sensory innervation of both sexes at early developmental stages. Subsequently, in males, androgens promote mammary mesenchymal expression of a truncated form of TrkB, which prevents BDNF-TrkB signaling in sensory axons and leads to a rapid loss of mammary gland innervation independent of neuronal apoptosis. Thus, sex hormone regulation of a neurotrophic factor signal directs sexually dimorphic axonal growth and maintenance, resulting in generation of a sex-specific neural circuit.