By prism-coupling an ordinary radiation mode into a lithium niobate planar waveguide, a second harmonic extraordinary guided mode can be generated if the proper coupling angle is chosen. By measuring this angle, the effective index of the radiation mode and thus of the interacting guided mode can be determined. This fact is exploited for the characterization of reverse-proton-exchanged waveguides, whose lowest order modes cannot be efficiently prism-coupled due to their confinement inside the sample.