The mammalian target of rapamycin (mTOR) is a key cell growth regulator, which forms two distinct functional complexes (mTORC1 and mTORC2). mTORC1, which is directly inhibited by rapamycin, promotes cell growth by stimulating protein synthesis and inhibiting autophagy. mTORC1 is regulated by a wide range of extra- and intracellular signals, including growth factors, nutrients, and energy levels. Precise regulation of mTORC1 is important for normal cellular physiology and development, and dysregulation of mTORC1 contributes to hypertrophy and tumorigenesis. In this study, we screened Drosophila small GTPases for their function in TORC1 regulation and found that TORC1 activity is regulated by members of the Rab and Arf family GTPases, which are key regulators of intracellular vesicle trafficking. In mammalian cells, uncontrolled activation of Rab5 and Arf1 strongly inhibit mTORC1 activity. Interestingly, the effect of Rab5 and Arf1 on mTORC1 is specific to amino acid stimulation, whereas glucose-induced mTORC1 activation is not blocked by Rab5 or Arf1. Similarly, active Rab5 selectively inhibits mTORC1 activation by Rag GTPases, which are involved in amino acid signaling, but does not inhibit the effect of Rheb, which directly binds and activates mTORC1. Our data demonstrate a key role of Rab and Arf family small GTPases and intracellular trafficking in mTORC1 activation, particularly in response to amino acids.