Lipid nanoparticles (LNPs) are widely recognized as crucial carriers of mRNA in therapeutic and vaccine development. The typical lipid composition of mRNA-LNP systems includes an ionizable lipid, a helper lipid, a polyethylene glycol (PEG)-lipid, and cholesterol. Concerns arise regarding cholesterol's susceptibility to oxidation, potentially leading to undesired immunological responses and toxicity. In this study, we formulated novel LNPs by replacing cholesterol with phytochemical-derived compounds, specifically ginsenoside Rg2 and its derivative phytosterol protopanaxadiol (PPD), and validated their efficacy as mRNA delivery systems. The mRNA-LNP complexes were manually prepared through a simple mixing process. The biocompatibility of these Rg2-based LNPs (Rg2-LNP) and PPD-based LNPs (PPD-LNP) was assessed through cell viability assays, while the protective function of LNPs for mRNA was demonstrated by RNase treatment. Enhanced green fluorescent protein (EGFP) mRNA delivery and expression in A549 and HeLa cells were analyzed using optical microscopy and flow cytometry. The expression efficiency of Rg2-LNP and PPD-LNP was compared with that of commercially available LNPs, with both novel formulations demonstrating superior transfection and EGFP expression. Furthermore, in vivo tests following intramuscular (I.M.) injection in hairless mice demonstrated efficient luciferase (Luc) mRNA delivery and effective Luc expression using Rg2-LNP and PPD-LNP compared to commercial LNPs. Results indicated that the efficiency of EGFP and Luc expression in Rg2-LNP and PPD-LNP surpassed that of the cholesterol-based LNP formulation. These findings suggest that Rg2-LNP and PPD-LNP are promising candidates for future drug and gene delivery systems.