Our aim is to investigate the association of being physically active on the prevalence of Type 2 diabetes mellitus among elderly people. We enrolled 53 men and 97 women, aged 65 to 100, from various areas of Cyprus. Physical activity was evaluated through a validated questionnaire (the short International Physical Activity Questionnaire, IPAQ). Prevalence of diabetes was 26% in men and 18% in women, while 55% of men and 50% of women were reported as being moderately or vigorously active. Furthermore, 5% of the participants reported that they smoked, 4% had stopped smoking, and 8% reported alcohol consumption. People in the upper tertile of the IPAQ score were 0.26 times less likely to have diabetes (p < 0.05); on the other hand people in the lower tertile of the score were 1.7 times more likely to have diabetes (p < 0.05). Our findings support the notion of a beneficial effect of physical activity on the burden of diabetes in the elderly.