Cell membranes have a complex lateral organization featuring domains with distinct composition, also known as rafts, which play an essential role in cellular processes such as signal transduction and protein trafficking. In vivo, perturbations of membrane domains (e.g., by drugs or lipophilic compounds) have major effects on the activity of raft-associated proteins and on signaling pathways, but they are difficult to characterize because of the small size of the domains, typically below optical resolution. Model membranes, instead, can show macroscopic phase separation between liquid-ordered and liquid-disordered domains, and they are often used to investigate the driving forces of membrane lateral organization. Studies in model membranes have shown that some lipophilic compounds perturb membrane domains, but it is not clear which chemical and physical properties determine domain perturbation. The mechanisms of domain stabilization and destabilization are also unknown. Here we describe the effect of six simple hydrophobic compounds on the lateral organization of phase-separated model membranes consisting of saturated and unsaturated phospholipids and cholesterol. Using molecular simulations, we identify two groups of molecules with distinct behavior: aliphatic compounds promote lipid mixing by distributing at the interface between liquid-ordered and liquid-disordered domains; aromatic compounds, instead, stabilize phase separation by partitioning into liquid-disordered domains and excluding cholesterol from the disordered domains. We predict that relatively small concentrations of hydrophobic species can have a broad impact on domain stability in model systems, which suggests possible mechanisms of action for hydrophobic compounds in vivo.