The relationship between cannabis and cognitive performance is controversial. While both acute administration and long-term cannabis use impair cognitive performance in healthy subjects, several studies have shown improved cognitive outcomes in patients with schizophrenia spectrum disorders who use cannabis. The aim of this study was to determine the relationship between lifetime cannabis use, as assessed longitudinally over 10 years of follow-up in a sample of 42 patients and 35 of their unaffected siblings, and current cognitive performance. Forty-two healthy control subjects were assessed at follow-up with the same instruments. Stepwise linear regression revealed a negative effect of longitudinal cannabis use on performance in a social cognition task in the patient group. In the sibling group, lifetime cannabis use had a negative effect on processing speed and declarative memory performance. In the control group, cannabis use per se did not predict cognitive performance; however, when adding lifetime tobacco use to the model, we found a negative association between lifetime cannabis and tobacco use and processing speed and social cognition performance. Moreover, a lower IQ associated with current cannabis use predicted worse attentional performance in the control group. The differential pattern of associations between cannabis use and cognitive performance in patients compared with siblings and controls can be explained by the negative impact of illness on cognition.