Controlling four-wave mixing (FWM) is vital for several applications, including fiber optical communication, optical signal processing, optical amplification, and frequency generation. This paper presents a novel, to our knowledge, approach to control unidirectional FWM in elliptically birefringent fibers. By leveraging the frequency-dependent polarization eigenmodes of these fibers and detuning the optical frequency of one of the pump fields by a few megahertz, we can turn the FWM interaction on and off, thus controlling the generation of signal and idler fields. Moreover, this approach allows us to turn off the FWM interaction at any desired frequency, enabling all-optical switching and narrowband filtering applications.