Development of a fundamental understanding of how peptides specifically interact with inorganic material surfaces is crucial to furthering many applications in the field of nanobiotechnology. Herein, we report systematic study of peptide sequence-activity relationships for binding to II-VI semiconductors (CdS, CdSe, ZnS, ZnSe) and Au using a yeast surface display system, and we define criteria for tuning peptide affinity and specificity for these material surfaces. First, homohexapeptides of the 20 naturally occurring amino acids were engineered, expressed on yeast surface, and assayed for the ability to bind each material surface in order to define functional groups sufficient for binding. Histidine (H6) was able to mediate binding of yeast to the five materials studied, while tryptophan (W6), cysteine (C6), and methionine (M6) exhibited different levels of binding to single-crystalline ZnS and ZnSe and polycrystalline Au surfaces. The ability of neighboring amino acids to up- and down-modulate histidine binding was then evaluated by use of interdigitated peptides (XHXHXHX). While the 20 amino acids exhibited a unique fingerprint of modulation for each material, some general trends emerged. With neutral defined by alanine, up-modulation occurred with glycine, basic amino acids, and the previously defined binding amino acids histidine, tryptophan, cysteine, and methionine, and down-modulation generally occurred with acidic, polar, and hydrophobic residues. We conclude that certain amino acids directly bind the material surface while neighboring amino acids locally modulate the binding environment for the materials we studied. Therefore, by the specific placement of up- and down-modulating amino acids, material specificity can be controlled. Finally, by employing the compositional and spatial criteria developed herein, it was possible to predictively design peptide sequences with material specificity, including a multimaterial binder, a Au-specific binder, and a ZnS-specific binder, that were verified as such in the context of yeast display.