The magnon thermal conductivity kappa(mag) of the hole-doped spin ladders in Sr14-xCaxCu24O41 has been investigated at low doping levels x. The analysis of kappa(mag) reveals a strong doping and temperature dependence of the magnon mean free path l(mag), which is a local probe for the interaction of magnons with the doped holes in the ladders. In particular, this novel approach to studying charge degrees of freedom via spin excitations shows that charge ordering of the holes in the ladders leads to a freezing out of magnon-hole scattering processes.