We demonstrate that the onset of complex spin orders in ACr2O4 spinels with magnetic and Jahn-Teller active A=Fe and Cu ions lowers the lattice symmetry. This is clearly indicated by the emergence of anisotropic lattice dynamics-i.e., by the pronounced phonon splittings-even when experiments probing static distortions fail. The crystal symmetry in the magnetic phase is reduced from tetragonal to orthorhombic for both compounds. The conical spin ordering in FeCr2O4 is also manifested in the hardening of the phonon frequencies. In contrast, the multiferroic CoCr2O4 with no orbital degrees of freedom shows tiny deviations from cubic structure even in its ground state.