Diabetes is thought to contribute to cognitive impairment in the elderly. The risk may be modified by genetic factors such as apolipoprotein E (APOE). We aim to determine the prevalence of dementia in Chinese diabetics and non-diabetic individuals, and assess the effect of the APOE genotype. A cross-sectional study was conducted among participants aged 50 and over in an urban community. Subjects were 1:1 matched for age and gender (diabetes group (n = 497) versus non-diabetes group (n = 497)). Each subject was interviewed for dementia and related risk factors. Fasting blood samples were drawn for glucose and APOE. Subjects were screened using the Mini Mental State Examination (MMSE) and were examined by a series of neuropsychological tests if screened positive (indicated by an MMSE score below 19, 21, and 24, respectively according to different educational attainment). A diagnosis of dementia was defined according to DSM-IV criteria. The prevalence of all-cause dementia, Alzheimer's disease, and vascular dementia in diabetics (4.8%, 2.7%, 1.4%) was higher than that of non-diabetics (2.2%, 1.2%, 0.4%) (all-cause dementia, diabetics versus non-diabetics, p < 0.05). APOE ε4 carriers had higher prevalence of dementia and AD (diabetics: 9.2%, 7.55%, non-diabetics: 3.3%, 3.3%) than APOE ε4 non-carriers (diabetics: 6.3%, 2.3%, non-diabetics: 2.35%, 0.7%) (p < 0.05). Diabetics carrying the APOE ε4 allele had a 3.982-fold [95%CI, 1.418-11.184] likelihood of having dementia compared with those without these two factors. This study showed that the prevalence of dementia was significantly higher in diabetics than non-diabetics. APOE ε4 further enhanced the risk.