We tested the specularity of the ref lection of slow atoms from an evanescent-wave mirror at normal incidence. In two of the three prisms that we tested the atoms were ref lected diffusely. This nonspecular ref lection appears to be correlated with the rms roughness of the surface supporting the evanescent wave. Only the highest quality surface (rms roughness of the order of 0.1 nm) leads to specular ref lection. This discovery imposes stringent limits on the use of these mirrors in atomic-optics experiments.