Phosphatidylinositol phosphates (PIPs) and cholesterol are known to regulate the function of late endosomes and lysosomes (LELs), and ORP1L specifically localizes to LELs. Here, we show in vitro that ORP1 is a PI(4,5)P2- or PI(3,4)P2-dependent cholesterol transporter, but cannot transport any PIPs. In cells, both ORP1L and PI(3,4)P2 are required for the efficient removal of cholesterol from LELs. Structures of the lipid-binding domain of ORP1 (ORP1-ORD) in complex with cholesterol or PI(4,5)P2 display open conformations essential for ORP function. PI(4,5)P2/PI(3,4)P2 can facilitate ORP1-mediated cholesterol transport by promoting membrane targeting and cholesterol extraction. Thus, our work unveils a distinct mechanism by which PIPs may allosterically enhance OSBP/ORPs-mediated transport of major lipid species such as cholesterol.