We experimentally investigate the dynamical instability of a Bose-Einstein condensate in an optical ring resonator for various cavity detuning and pump powers. The resulting stability diagram is asymmetric with respect to the cavity detuning and can be described by the coupling of two atomic modes with one optical mode. We compare the experimental data to a numerical simulation and to an analytic expression of the phase boundary. For positive and negative pump cavity detuning, different coupling mechanisms are identified explaining the asymmetry of the stability diagram. We present a physical interpretation and discuss the connection to the Dicke quantum phase transition.