We previously established that vascular smooth muscle-derived adventitial progenitor cells (AdvSca1-SM) preferentially differentiate into myofibroblasts and contribute to fibrosis in response to acute vascular injury. However, the role of these progenitor cells in chronic atherosclerosis has not been defined. Using an AdvSca1-SM lineage tracing model, scRNA-Seq, flow cytometry, and histological approaches, we confirmed that AdvSca1-SM cells localize throughout the vessel wall and atherosclerotic plaques, where they primarily differentiate into fibroblasts, SMCs, or remain in a stem-like state. Klf4 knockout specifically in AdvSca1-SM cells induced transition to a more collagen-enriched myofibroblast phenotype compared to WT mice. Additionally, Klf4 depletion drastically modified the phenotypes of non-AdvSca1-SM-derived cells, resulting in more contractile SMCs and atheroprotective macrophages. Functionally, overall plaque burden was not altered with Klf4 depletion, but multiple indices of plaque vulnerability were reduced. Collectively, these data support that modulating the AdvSca1-SM population confers increased protection from the development of unstable atherosclerotic plaques.