This study aims to analyze the association between perceived characteristics of the neighborhood environment and changes in leisure-time physical activity in Brazilian older adults. Longitudinal analysis was performed considering a population-based study carried out in 2009/2010 with follow-up in 2013/2014. Changes in leisure-time physical activity, obtained with the International Physical Activity Questionnaire during both waves, were associated with data of environment perception with the Neighborhood Environment Walkability Scale - Abbreviated (baseline only) performing multinomial logistic regression. In total, 1,162 older adults (65.2% women, mean age = 73.7 years) participated. Older adults who perceived flat streets, bicycle paths, and high flow of vehicles in the neighborhood were more likely to remain active in leisure time walking. Those who noticed the presence of flat streets also had a greater chance of becoming active than those who did not notice it. Older adults who perceived bicycle paths and sports events were more likely to maintain moderate and vigorous physical activity during leisure time. Furthermore, older adults who noticed parks and squares close to their residence were 0.49 times less likely to become insufficiently active. Brazilian older adults are more active in neighborhoods with more favorable attributes regarding leisure-time physical activity.