The complete active space self-consistent field (CASSCF) method is a cornerstone in modern excited-state quantum chemistry providing the starting point for most common multireference computations. However, CASSCF, when used with a minimal active space, can produce significant errors (>2 eV) even for the excitation energies of simple hydrocarbons if the states of interest possess ionic character. After illustrating this problem in some detail, we present a diagnostic for ionic character, denoted as Q at, that is readily computed from the transition density. A set of 11 molecules is considered to study errors in vertical excitation energies. State-averaged CASSCF obtains a mean absolute error (MAE) of 0.87 eV for the 34 singlet states considered. We highlight a strong correlation between the obtained errors and the Q at diagnostic, illustrating its power to predict problematic cases. Conversely, using multireference configuration interaction with single and double excitations and Pople's size extensivity correction (MR-CISD+P), excellent results are obtained with an MAE of 0.11 eV. Furthermore, correlations with the Q at diagnostic disappear. In summary, we hope that the presented diagnostic will facilitate reliable and user-friendly multireference computations on conjugated organic molecules.