The Goos-Hänchen (GH) shift and the Imbert-Fedorov (IF) shift are optical phenomena which describe the longitudinal and transverse lateral shifts at the reflection interface, respectively. Here, we predict the GH and IF shifts in Weyl semimetals (WSMs)-a promising material harboring low energy Weyl fermions, a massless fermionic cousin of photons. Our results show that the GH shift in WSMs is valley independent, which is analogous to that discovered in a 2D relativistic material-graphene. However, the IF shift has never been explored in nonoptical systems, and here we show that it is valley dependent. Furthermore, we find that the IF shift actually originates from the topological effect of the system. Experimentally, the topological IF shift can be utilized to characterize the Weyl semimetals, design valleytronic devices of high efficiency, and measure the Berry curvature.