The Eps homology (EH) domain is a recently described protein binding module that is found, in multiple or single copies, in several proteins in species as diverse as human and yeast. In this work, we have investigated the molecular details of recognition specificity mediated by this domain family by characterizing the peptide-binding preference of 11 different EH domains from mammal and yeast proteins. Ten of the eleven EH domains could bind at least some peptides containing an Asn-Pro-Phe (NPF) motif. By contrast, the first EH domain of End3p preferentially binds peptides containing an His-Thr/Ser-Phe (HT/SF) motif. Domains that have a low affinity for the majority of NPF peptides reveal some affinity for a third class of peptides that contains two consecutive amino acids with aromatic side chains (FW or WW). This is the case for the third EH domain of Eps15 and for the two N-terminal domains of YBL47c. The consensus sequences derived from the peptides selected from phage-displayed peptide libraries allows for grouping of EH domains into families that are characterized by different NPF-context preference. Finally, comparison of the primary sequence of EH domains with similar or divergent specificity identifies a residue at position +3 following a conserved tryptophan, whose chemical characteristics modulate binding preference.