Human adipose tissue (hAT) is constituted of structural units termed lobules, the organization of which remains to be defined. Here we report that lobules are composed of two extracellular matrix compartments, i.e., septa and stroma, delineating niches of CD45-/CD34+/CD31- progenitor subsets characterized by MSCA1 (ALPL) and CD271 (NGFR) expression. MSCA1+ adipogenic subset is enriched in stroma while septa contains mainly MSCA1-/CD271- and MSCA1-/CD271high progenitors. CD271 marks myofibroblast precursors and NGF ligand activation is a molecular relay of TGFβ-induced myofibroblast conversion. In human subcutaneous (SC) and visceral (VS) AT, the progenitor subset repartition is different, modulated by obesity and in favor of adipocyte and myofibroblast fate, respectively. Lobules exhibit depot-specific architecture with marked fibrous septa containing mesothelial-like progenitor cells in VSAT. Thus, the human AT lobule organization in specific progenitor subset domains defines the fat depot intrinsic capacity to remodel and may contribute to obesity-associated cardiometabolic risks.