The dependence of the near-field signal on the dielectric function of a specific material proposes scattering-type near-field optical microscopy (s-SNOM) as a viable tool for material characterization studies. Our experiment shows that specific material identification by s-SNOM is not a straightforward task as parameters involved in the detection scheme can also influence material contrast measurements. More precisely, we demonstrate that s-SNOM contrast in a pseudo-heterodyne detection configuration depends on the oscillation amplitude of the reference mirror and that for reliable measurements of the contrast between different materials this aspect needs to be taken into consideration.