The target of rapamycin (TOR) protein is a conserved regulator of ribosome biogenesis, an important process for cell growth and proliferation. However, how TOR is involved remains poorly understood. In this study, we find that rapamycin and nutrient starvation, conditions inhibiting TOR, lead to significant nucleolar size reduction in both yeast and mammalian cells. In yeast, this morphological change is accompanied by release of RNA polymerase I (Pol I) from the nucleolus and inhibition of ribosomal DNA (rDNA) transcription. We also present evidence that TOR regulates association of Rpd3-Sin3 histone deacetylase (HDAC) with rDNA chromatin, leading to site-specific deacetylation of histone H4. Moreover, histone H4 hypoacetylation mutations cause nucleolar size reduction and Pol I delocalization, while rpd3Delta and histone H4 hyperacetylation mutations block the nucleolar changes as a result of TOR inhibition. Taken together, our results suggest a chromatin-mediated mechanism by which TOR modulates nucleolar structure, RNA Pol I localization and rRNA gene expression in response to nutrient availability.