Binding specificity is an important consideration in drug design. An effective drug molecule often must bind with high specificity to its intended target in the body; lower specificity implies the possibility of significant binding to unintended partners, which could instigate deleterious side effects. However, if the target is a rapidly mutating agent, a drug that is too specific will quickly lose its efficacy by not binding well to functional mutants. Therefore, in molecular design, it is crucial to tailor the binding specificity of a drug to the problem at hand. In practice, specificity is often studied on a case-by-case basis, and it is difficult to create general understanding of the determinants of specificity from the union of such available cases. In this work, we undertook a comprehensive, general study of molecular binding with emphasis on understanding the determinants of specificity from a physical standpoint. By extending a theoretical framework grounded in continuum electrostatics and creating an abstracted lattice model that captures key physical aspects of binding interactions, we systematically explored the relationship between a molecule's physical characteristics and its binding specificity toward potential partners. The theory and simulated binding interactions suggested that charged molecules are more specific binders than their hydrophobic counterparts for several reasons. First, the biological spectrum of possible binding characteristics includes more partners that bind equally well to hydrophobic ligands than to charged ligands. Also, charged ligands, whose electrostatic potentials have strong orientational dependence, are more sensitive to shape complementarity than their hydrophobic counterparts. Ligand conformational and orientational flexibility can further influence a charged molecule's ability to bind specifically. Interestingly, we found that conformational flexibility can increase the specificity of polar and charged ligands, by allowing them to greatly lower the binding free energy to a select few partners relative to others. Additionally, factors such as a molecule's size and the ionic strength of the solution were found to predictably affect binding specificity. Taken together, these results, all of which stem from a unified theoretical framework, provide valuable physical insight into the general determinants of binding specificity and promiscuity in a biological environment. The general principles discussed here could prove useful in the design of molecules with tailored specificities, leading to more effective therapeutics.