The isovector and isotensor energy differences between yrast states of isobaric multiplets in the lower half of the pf region are quantitatively reproduced in a shell model context. The isospin nonconserving nuclear interactions are found to be at least as important as the Coulomb potential. Their isovector and isotensor channels are dominated by J=2 and J=0 pairing terms, respectively. The results are sensitive to the radii of the states, whose evolution along the yrast band can be accurately followed.