Humans share with other animals a system for thinking about numbers in an imprecise and intuitive way. The Approximate Number System (ANS) that underlies this thinking is present throughout the lifespan, is entirely nonverbal, and supports basic numerical computations like comparing, adding, and subtracting quantities. Humans, unlike other animals, also have a system for representing exact numbers. This linguistically mediated system is slowly mastered over the course of many years and provides the basis for most of our formal mathematical thought. A growing body of evidence suggests that the nonverbal ANS and the culturally invented system of exact numbers are fundamentally linked. In this article, we review evidence for this relationship, describing how group and individual differences in the ANS correlate with and even predict formal math ability. In this way, we illustrate how a system of ancient core knowledge may serve as a foundation for more complex mathematical thought.