We describe a first-principles, easy-to-implement, and efficient approach for determining the structural geometry of insulating solids under finite electric fields. This method consists of simultaneously minimizing the field-induced total ionic forces and the electric free energy. Moreover, we present a theory to analyze its predictions that provides a microscopic understanding of electro-mechanical responses in materials. We illustrate this approach by computing piezoelectric and dielectric responses of two rather different compounds, namely, ferroelectric PbTiO3 and semiconductor GaN.