The insulin signaling pathway is critical in regulating glucose levels and is associated with diabetes, obesity, and longevity. A tyrosine phosphorylation cascade creates docking sites for protein interactions, initiating subsequent propagation of the signal throughout the cell. The phosphotyrosine interactome of this medically important pathway has not yet been studied comprehensively. We therefore applied quantitative interaction proteomics to exhaustively profile all potential phosphotyrosine-dependent interaction sites in its key players. We targeted and compared insulin receptor substrates 1 and 2 (IRS-1 and IRS-2) as central distributors of the insulin signal, the insulin receptor, the insulin-like growth factor 1 receptor, and the insulin receptor-related receptor. Using the stable isotope labeling by amino acids in cell culture (SILAC) approach with phosphorylated versus non-phosphorylated bait peptides, we found phosphorylation-specific interaction partners for 52 out of 109 investigated sites. In addition, doubly and triply phosphorylated motifs provided insight into the combinatorial effects of phosphorylation events in close proximity to each other. Our results retrieve known interactions and substantially broaden the spectrum of potential interaction partners of IRS-1 and IRS-2. A large number of common interactors rationalize their extensive functional redundancy. However, several proteins involved in signaling and metabolism interact differentially with IRS-1 and IRS-2 and thus provide leads into their different physiological roles. Differences in interactions at the receptor level are reflected in multisite recruitment of SHP2 by the insulin-like growth factor 1 receptor and limited but exclusive interactions with the IRR. In common with other recent reports, our data furthermore hint at non-SH2 or phosphotyrosine-binding domain-mediated phosphotyrosine binding.