This work presents a theoretical analysis of the motion of a tracer colloid driven by a time-dependent force through a viscoelastic fluid. The recoil of the colloid after application of a strong force is determined. It provides insights into the elastic forces stored locally in the fluid and their weakening by plastic processes. We generalize the mode-coupling theory of microrheology to include time-dependent forces. After deriving the equations of motion for the tracer correlator and simplifying to a schematic model, we apply the theory to a switch-off force protocol that features the recoiling of the tracer after cessation of the driving. We also include Langevin dynamics simulations to compare to the results of the theory. A nonmonotonic trend of the recoil amplitude is found in the theory and confirmed in the simulations. The linear-response approximation is also verified in the small-force regime. While the overall agreement between simulation and theory is good, simulation shows that the theory predicts a too strong nonmonotonous dependence of the recoil distance on the applied force.