We investigated the ab-plane absorption spectra of RMnO3 (R=La, Pr, Nd, Gd, and Tb) thin films. As the ionic radius of the R ion decreases, we observed a drastic suppression of the 2 eV peak, i.e., the intersite optical transition between spin- and orbital-aligned states across the Mott gap. We found that, in addition to orbital rotation, orbital mixing in the orbital-ordered state should play an important role in the suppression of 2 eV peak. We also found that the spectral weight of 2 eV peak is proportional to the A-type antiferromagnetic ordering temperature, which suggests that the magnetic interaction should be sensitively coupled to the orbital degree of freedom.