The overexpression of two arginase (ARG) isoforms, ARG1 and ARG2, contributes to the onset of numerous disorders, including cardiovascular and immune-mediated diseases, as well as tumors. To elucidate the specific roles of ARG1 and ARG2 without interfering with their physiological functions, it is crucial to develop effective ARG inhibitors that target only one isoform, while maintaining low toxicity and an adequate pharmacokinetic profile. In this context, we present a comprehensive overview of the different generations of ARG inhibitors. Given the general lack of selectivity in most existing inhibitors, we analyzed the structural features and plasticity of the ARG1 and ARG2 binding sites to explore the potential for designing inhibitors with novel binding patterns. We also review ongoing preclinical and clinical studies on selected inhibitors, highlighting both progress and challenges in developing potent, selective ARG inhibitors. Furthermore, we discuss medicinal chemistry strategies that may accelerate the discovery of selective ARG inhibitors.