The canonical AD pathological cascade posits that the accumulation of amyloid beta ( ) is the initiating event, accelerating the accumulation of tau in the entorhinal cortex (EC), which subsequently spreads into the neocortex. Here in a sample of over 1300 participants with multimodal imaging and genetic information we queried how genetic variation affects these stages of the AD cascade. We observed that females and homozygotes are more susceptible to the effects of on the primary accumulation of tau, with greater EC tau for a given level of . Furthermore, we observed for individuals who have rare risk variants in Triggering Receptor Expressed on Myeloid Cells 2 (TREM2) and/or homozygotes there was a greater spread of primary tau from the EC into the neocortex. These findings offer insights into the function of sex, APOE and microglia in AD progression, and have implications for determining personalised treatment with drugs targeting and tau.