Neuroanatomical variation in individuals with bipolar disorder (BD) has been previously described in observational studies. However, the causal dynamics of these relationships remain unexplored. We performed Mendelian Randomization of 297 structural and functional neuroimaging phenotypes from the UK BioBank and BD using genome-wide association study summary statistics. We found 28 significant causal relationship pairs after multiple testing corrections containing BD as a term, 27 of which described neuroimaging phenotype effects on BD. We applied an inverse sparse regression algorithm to estimate the direct effect of phenotypes conditional on all other causal effects, finding that white matter tract phenotypes have larger absolute effects on BD than vice versa. We found that white matter phenotypes have significantly larger out-degrees than non-white matter tract phenotypes, and that the effect of neuroimaging variation on BD is larger than vice versa. Our results provide support for the hypothesis that neuroanatomical variation, specifically in white matter tracts such as the superior and inferior longitudinal fasciculi, is a cause rather than a consequence of BD.