Fiber tractography in baby diffusion MRI is challenging due to the low and spatially-varying diffusion anisotropy, causing most tractography algorithms to yield streamlines that fall short of reaching the cortex. In this paper, we introduce a method called asymmetry spectrum imaging (ASI) to improve the estimation of white matter pathways in the baby brain by (i) incorporating an asymmetric fiber orientation model to resolve subvoxel fiber configurations such as fanning and bending, and (ii) explicitly modeling the range (or spectrum) of typical diffusion length scales in the developing brain. We validated ASI using in-vivo baby diffusion MRI data from the Baby Connectome Project (BCP), demonstrating that ASI can characterize complex subvoxel fiber configurations and accurately estimate the fiber orientation distribution function in spite of changes in diffusion patterns. This, in turn, results in significantly better diffusion tractography in the baby brain.