We investigate the thermal and Kerr nonlinearity in a system of two optically-coupled silica microtoroid resonators experimentally and theoretically. A model for two coupled oscillators describing nonlinear resonance curves is developed. Stability of the static solutions is analyzed. It is shown that thermal nonlinearity is responsible for driving the eigenfrequencies of the two resonators apart, making the normal modes of the system unstable as the pump power grows. The red-detuned normal mode becomes unstable for certain pumping powers.