Hypoxia-inducible factor-2 (HIF-2) is a heterodimeric transcription factor formed through dimerization between an oxygen-sensitive HIF-2α subunit and its obligate partner subunit ARNT. Enhanced HIF-2 activity drives some cancers, whereas reduced activity causes anemia in chronic kidney disease. Therefore, modulation of HIF-2 activity via direct-binding ligands could provide many new therapeutic benefits. Here, we explored HIF-2α chemical ligands using combined crystallographic, biophysical, and cell-based functional studies. We found chemically unrelated antagonists to employ the same mechanism of action. Their binding displaced residue M252 from inside the HIF-2α PAS-B pocket toward the ARNT subunit to weaken heterodimerization. We also identified first-in-class HIF-2α agonists and found that they significantly displaced pocket residue Y281. Its dramatic side chain movement increases heterodimerization stability and transcriptional activity. Our findings show that despite binding to the same HIF-2α PAS-B pocket, ligands can manifest as inhibitors versus activators by mobilizing different pocket residues to allosterically alter HIF-2α-ARNT heterodimerization.