State-of-the-art ultrafast mid-IR fiber lasers deliver optical solitons with durations of several hundred femtoseconds. The Er- or Ho-doped fluoride gain fibers generally used in these lasers have strong anomalous dispersion at ∼3 µm, which generally forces them to operate in the soliton regime. Here we report that a pulse-energy clamping effect, caused by the buildup of intracavity nonlinearities, limits the shortest obtainable pulse durations in these mid-infrared soliton fiber lasers. Excessive intra-cavity energy results in soliton instability, collapse and fragmentation into a variety of stable multi-pulse states, including phase-locked soliton molecules and harmonically mode-locked states. We report that the spectral evolution of the mid-IR laser pulses can be recorded between roundtrips through stretching their second-harmonic signal in a 25-km-length of single-mode fiber. Using a modified dispersive Fourier transform set-up, we were able to perform for the first time spectro-temporal measurements of mid-IR laser pulses both in the pulsed state and during pulse collapse and fragmentation. The results provide insight into the complex nonlinear dynamics of mid-IR soliton fiber lasers and open up new opportunities for obtaining a variety of stable multi-pulse mode-locked states at mid-IR wavelengths.