Charge-carrier traps play a central role in the excited-state dynamics of semiconductor nanocrystals, but their influence is often difficult to measure directly. In CdS and CdSe nanorods of nonuniform width, spatially separated electrons and trapped holes display relaxation dynamics that follow a power-law function in time that is consistent with a recombination process limited by trapped-hole diffusion. However, power-law relaxation can originate from mechanisms other than diffusion. Here we report transient absorption spectroscopy measurements on CdS and CdSe nanorods recorded at temperatures ranging from 160 to 294 K. We find that the exponent of the power law is temperature-independent, which rules out several models based on stochastic activated processes and provides insights into the mechanism of diffusion-limited recombination in these structures. The data point to weak electronic coupling between trap states and suggest that surface-localized trapped holes couple strongly to phonons, leading to slow diffusion. Trap-to-trap hole hopping behaves classically near room temperature, while quantum aspects of phonon-assisted tunneling become observable at low temperatures.