Sensitive detection of a tumor marker, neuron-specific enolase (NSE), was performed by a label-free direct immunoassay based on a colorimetric plasmonic biosensor. Reflective plasmonic colors of silver nanodome arrays provided a way for a sensitive refractive index sensor based on spectrometer-free colorimetric detection. The direct detection of NSE was demonstrated by a combination of a sensitive sensor substrate and image processing. The limit of detection (LOD) for NSE was determined to be 270 pM, which is lower than the clinical threshold value of NSE used for medical diagnostics of small-cell lung cancer. Since our substrate-based colorimetric plasmonic biosensor is compatible with smartphone detection, we believe that the presented biosensor will open up a way for biosensor technology for point-of-care testing as well as mobile health applications.