In contemporary society, it is essential to have adequate mathematical skills. Being numerate has been linked to positive life outcomes and well-being in adults. It is also acknowledged that math anxiety (MA) hampers mathematical skills increasingly with age. Still, the mechanisms by which MA affect performance remain debated. Using structural equation modeling (SEM), we contrast the different ways in which MA has been suggested to interfere with math abilities. Our models indicate that MA may affect math performance through three pathways: (1) indirectly through working memory ability, giving support for the 'affective drop' hypothesis of MA's role in mathematical performance, (2) indirectly through symbolic number processing, corroborating the notion of domain-specific mechanisms pertaining to number, and (3) a direct effect of MA on math performance. Importantly, the pathways vary in terms of their relative strength depending on what type of mathematical problems are being solved. These findings shed light on the mechanisms by which MA may interfere with mathematical performance.