The crossover from orthogonal to the unitary universality classes in the distribution of the anomalously localized states (ALS) in two-dimensional disordered conductors is traced as a function of magnetic field. We demonstrate that the microscopic origin of the crossover is the change in the symmetry of the underlying disorder configurations that are responsible for ALS. These disorder configurations are of weak magnitude (compared to the Fermi energy) and of small size (compared to the mean free path). We find their shape explicitly by means of the direct optimal fluctuation method.