Phonons and magnons are engineered by periodic potential landscapes in phononic and magnonic crystals, and their combined studies may enable valley phonon transport tunable by the magnetic field. Through nonreciprocal surface acoustic wave transmission, we demonstrate valley-selective phonon-magnon scattering in magnetoelastic superlattices. The lattice symmetry and the out-of-plane magnetization component control the sign of nonreciprocity. The phonons in the valleys play a crucial role in generating nonreciprocal transmission by inducing circularly polarized strains that couple with the magnons. The transmission spectra show a nonreciprocity peak near a transmission gap, matching the phononic band structure. Our results open the way for manipulating valley phonon transport through periodically varying magnon-phonon coupling.