The orphan nuclear hormone receptor liver receptor homologous protein-1 (LRH-1; NR5A2, also known as FTF), an unusual receptor that binds DNA as a monomer, is an essential regulator of expression of a rate-limiting enzyme in bile acid formation, cholesterol 7-alpha-hydroxylase. In a classic negative feedback loop that is a crucial component of the complex regulation of cholesterol metabolism, cholesterol 7-alpha-hydroxylase expression is decreased when bile acid levels are high. This repression is thought to be based on the bile acid-dependent induction of expression of the orphan receptor small heterodimer partner (SHP) NR0B2, which inhibits the activity of LRH-1. We have explored the molecular basis for this important regulatory effect by characterizing the mechanisms by which mouse and human SHP inhibit LRH-1-mediated transactivation. Both SHP proteins specifically interact with the AF-2 transactivation domain of LRH-1 both in vivo and in vitro. This domain is a common target for coactivator interaction, and the SHP proteins can compete with p160 coactivators for binding to LRH-1. In addition to the N-terminal receptor interaction domain, SHP includes a C-terminal domain with autonomous repression function. Neither a deletion nor a point mutation specifically affecting this domain blocked the ability to interact with LRH-1 to compete for coactivator binding or to repress LRH-1 transactivation. However, the relative ability of these mutants to inhibit LRH-1-mediated transactivation was markedly decreased. We conclude that the proposed central role of SHP in cholesterol metabolism is based on a two-step mechanism that is dependent on both coactivator competition and direct transcriptional repression.