The precision of a quantum sensor can overcome its classical counterpart when its constituents are entangled. In Gaussian squeezed states, quantum correlations lead to a reduction of the quantum projection noise below the shot noise limit. However, the most sensitive states involve complex non-Gaussian quantum fluctuations, making the required measurement protocol challenging. Here we measure the sensitivity of nonclassical states of the electronic spin J=8 of dysprosium atoms, created using light-induced nonlinear spin coupling. Magnetic sublevel resolution enables us to reach the optimal sensitivity of non-Gaussian (oversqueezed) states, well above the capability of squeezed states and about half the Heisenberg limit.