mRNA-based therapies hold tremendous promise for treating various diseases, yet their clinical success is hindered by delivery challenges. This study developed a library of 140 lipocationic Poly(β-amino ester)s (PBAEs) and formulated lipid-polymer hybrid nanoparticles (LPHs) with four helper lipids, including 1,2-dioleoyl-3-trimethylammonium-propane (DOTAP), to enhance mRNA delivery. Initial in vitro screening of four representative PBAEs identified the D/P4-1 formulation (DOTAP/PBAE molar ratio of 4:1) as the most effective. Further screening of the library using this formulation identified eight top-performing LPHs. In vivo experiments confirmed high luciferase expression in the spleen and lungs of mice following intravenous administration of Luc mRNA-loaded LPHs. Detailed analysis revealed that DOTAP incorporation influenced LPH properties, including apparent pKa, surface charge, and internal hydrophobicity, enabling enhanced mRNA release and cellular uptake. This study demonstrates potent approaches to modulate PBAE-lipid nanoparticle properties by altering PBAE structures and nanoparticle composition, offering insights for designing effective hybrid carriers for mRNA therapeutics.