In a classical description, a neutral, polarizable object acquires a kinetic momentum when exposed to crossed electric and magnetic fields. In the presence of only a magnetic field no such momentum exists classically, although it is symmetry allowed for an object lacking mirror symmetry. We perform a full QED calculation to show that the quantum vacuum coupled to a chiral molecule provides it with a kinetic "Casimir" momentum directed along the magnetic field, and proportional to its molecular rotatory power and to the fine structure constant.