In this work, an atomic layer deposited (ALD) Al2O3 ultrathin layer was introduced to passivate the ZnO-nanoparticle (NP) buffer layer of inverted polymer solar cells (PSCs) based on P3HT:PCBM. The surface morphology of the ZnO-NP/Al2O3 interface was systematically analyzed by using a variety of tools, in particular transmission electron microscopy (TEM), evidencing a conformal ALD-Al2O3 deposition. The thickness of the Al2O3 layers was optimized at the nanoscale to boost electron transport of the ZnO-NP layer, which can be attributed to the suppression of oxygen vacancy defects in ZnO-NPs confirmed by photoluminescence measurement. The optimal inverted PSCs passivated by ALD-Al2O3 exhibited an ∼22% higher power conversion efficiency than the control devices with a pristine ZnO-NP buffer layer. The employment of the ALD-Al2O3 passivation layer with precisely controlled thickness provides a promising approach to develop high efficiency PSCs with novel polymer materials.