Using magneto-optical microscopy in combination with ellipsometry measurements, we show that all-optical switching with polarized femtosecond laser pulses in ferrimagnetic GdFeCo is subjected to a threshold fluence absorbed in the magnetic layer, independent of either the excitation wavelength or the polarization of the laser pulse. Furthermore, we present a quantitative explanation of the intensity window in which all-optical helicity-dependent switching (AO-HDS) occurs, based on magnetic circular dichroism. This explanation is consistent with all the experimental findings on AO-HDS so far, varying from single- to multiple-shot experiments. The presented results give a solid understanding of the origin of AO-HDS, and give novel insights into the physics of ultrafast, laser controlled magnetism.