The aim of this study was to determine internal scale validity and reliability of P-Drive on-road by observing driving performance among people with neurological disorders. The study comprised a consecutive sample of 205 persons with stroke, dementia, or mild cognitive impairment who were remitted for and performed a standardized test in real traffic. Their driving abilities were evaluated with a new assessment tool, P-Drive on-road. Aspects of validity and reliability of the tool were analyzed using a Rasch rating scale model. The results indicated that the rating scale functioned as intended and a principal component analysis indicated that the items formed a unidimensional scale. Acceptable levels of person response validity were found and the scale was also able to distinguish between the abilities of the drivers with a person separation reliability of 0.90. The results indicate that P-Drive on-road is valid and reliable for producing a linear measure of driving ability in people with stroke, dementia and mild cognitive impairment, and has the potential to become a clinically useful assessment tool for on-road driving tests.