This paper describes theoretical and experimental investigations of an optical time domain reflectometry (OTDR) that conducts measurements in the 2-LP mode region of the fiber under test. Rayleigh backscattering amplitude coefficients from the HE11 mode to higher-order vector modes are analyzed to clarify the evolution in the state of polarization of the backscattered vector modes with respect to that of the probe HE11 mode. Based on the analysis, we then propose a technique for reducing amplitude fluctuations in the OTDR traces attributed to intensity distribution evolution of the LP11 mode. A proof-of-concept demonstration is performed on optical fibers in the laboratory and field environments, and the measured results confirm its usefulness.