Low levels of insulin-like growth factor-1 (IGF-1), an essential neurotrophic factor, have been associated with worse cognitive function in older adults. However, few studies have assessed the prospective association of serum IGF-1 with cognitive function. We aimed to determine the association between serum IGF-1 on concurrent and prospective cognitive function in a population sample of men aged 40-80 years. Blood samples were assessed for IGF-1 levels at baseline and neuropsychological assessments were performed at baseline (n = 400) and at follow-up after a mean duration of 8.3 years (n = 286). Linear regression analyses were carried out to determine the associations between quintiles of IGF-1 and cognitive function at the baseline and follow-up visits. Results showed that those in the top quintile of IGF-1 had lower processing capacity and global cognition scores at follow-up after controlling for cognitive function at baseline and other confounding factors. Additional analyses exploring associations with IGF-1 separately in middle-aged and older participants, and with quartiles of IGF-1 produced similar results. In those older than 60 years, high IGF-1 levels were also associated with lower baseline processing capacity. These results suggest that high IGF-1 levels are associated with worse long-term cognition in men. Together with past studies, we suggest that both, high and low levels of IGF-1 may be associated with poor cognitive function and that optimum levels of IGF-1 (quintile 2 and 3 in current study) may be associated with better cognitive function.