We combine tools from effective field theory and generalized unitarity to construct a map between on-shell scattering amplitudes and the classical potential for interacting spinless particles. For general relativity, we obtain analytic expressions for the classical potential of a binary black hole system at second order in the gravitational constant and all orders in velocity. Our results exactly match all known results up to fourth post-Newtonian order, and offer a simple check of future higher order calculations. By design, these methods should extend to higher orders in perturbation theory.