Mounting evidence indicates that the polygenic basis of late-onset Alzheimer's disease can be harnessed to identify individuals at greatest risk for cognitive decline. We have previously developed and validated a polygenic hazard score comprising of 31 single nucleotide polymorphisms for predicting Alzheimer's disease dementia age of onset. In this study, we examined whether polygenic hazard scores are associated with: (i) regional tracer uptake using amyloid PET; (ii) regional volume loss using longitudinal MRI; (iii) post-mortem regional amyloid-β protein and tau associated neurofibrillary tangles; and (iv) four common non-Alzheimer's pathologies. Even after accounting for APOE, we found a strong association between polygenic hazard scores and amyloid PET standard uptake volume ratio with the largest effects within frontal cortical regions in 980 older individuals across the disease spectrum, and longitudinal MRI volume loss within the entorhinal cortex in 607 older individuals across the disease spectrum. We also found that higher polygenic hazard scores were associated with greater rates of cognitive and clinical decline in 632 non-demented older individuals, even after controlling for APOE status, frontal amyloid PET and entorhinal cortex volume. In addition, the combined model that included polygenic hazard scores, frontal amyloid PET and entorhinal cortex volume resulted in a better fit compared to a model with only imaging markers. Neuropathologically, we found that polygenic hazard scores were associated with regional post-mortem amyloid load and neuronal neurofibrillary tangles, even after accounting for APOE, validating our imaging findings. Lastly, polygenic hazard scores were associated with Lewy body and cerebrovascular pathology. Beyond APOE, we show that in living subjects, polygenic hazard scores were associated with amyloid deposition and neurodegeneration in susceptible brain regions. Polygenic hazard scores may also be useful for the identification of individuals at the highest risk for developing multi-aetiological dementia.