Magnetization transfer ratio (MTR) and brain volumetric imaging are (semi-)quantitative MRI markers capturing demyelination, axonal degeneration and/or inflammation. However, factors shaping variation in these traits are largely unknown. In this study, we collected a longitudinal cohort of 33 multiple sclerosis (MS) patients and extended it cross-sectionally to 213. We measured MTR in lesions, normal-appearing white matter (NAWM), normal-appearing grey matter (NAGM) and total brain, grey matter, white matter and lesion volume. We also calculated the polygenic MS risk score. Longitudinally, inter-patient differences at inclusion and intra-patient changes during follow-up together explained > 70% of variance in MRI, with inter-patient differences at inclusion being the predominant source of variance. Cross-sectionally, we observed a moderate correlation of MTR between NAGM and NAWM and, less pronounced, with lesions. Age and gender explained about 30% of variance in total brain and grey matter volume. However, they contributed less than 10% to variance in MTR measures. There were no significant associations between MRI traits and the genetic risk score. In conclusion, (semi-)quantitative MRI traits change with ongoing disease activity but this change is modest in comparison to pre-existing inter-patient differences. These traits reflect individual variation in biological processes, which appear different from those involved in genetic MS susceptibility.