There is strong evidence for brain-related abnormalities in COVID-19 1-13 . It remains unknown however whether the impact of SARS-CoV-2 infection can be detected in milder cases, and whether this can reveal possible mechanisms contributing to brain pathology. Here, we investigated brain changes in 785 UK Biobank participants (aged 51-81) imaged twice, including 401 cases who tested positive for infection with SARS-CoV-2 between their two scans, with 141 days on average separating their diagnosis and second scan, and 384 controls. The availability of pre-infection imaging data reduces the likelihood of pre-existing risk factors being misinterpreted as disease effects. We identified significant longitudinal effects when comparing the two groups, including: (i) greater reduction in grey matter thickness and tissue-contrast in the orbitofrontal cortex and parahippocampal gyrus, (ii) greater changes in markers of tissue damage in regions functionally-connected to the primary olfactory cortex, and (iii) greater reduction in global brain size. The infected participants also showed on average larger cognitive decline between the two timepoints. Importantly, these imaging and cognitive longitudinal effects were still seen after excluding the 15 cases who had been hospitalised. These mainly limbic brain imaging results may be the in vivo hallmarks of a degenerative spread of the disease via olfactory pathways, of neuroinflammatory events, or of the loss of sensory input due to anosmia. Whether this deleterious impact can be partially reversed, or whether these effects will persist in the long term, remains to be investigated with additional follow up.