The soft part of the Earth's surface - the ground beneath our feet - constitutes the basis for life and natural resources, yet a general physical understanding of the ground is still lacking. In this critical time of climate change, cross-pollination of scientific approaches is urgently needed to better understand the behavior of our planet's surface. The major topics in current research in this area cross different disciplines, spanning geosciences, and various aspects of engineering, material sciences, physics, chemistry, and biology. Among these, soft matter physics has emerged as a fundamental nexus connecting and underpinning many research questions. This perspective article is a multi-voice effort to bring together different views and approaches, questions and insights, from researchers that work in this emerging area, the soft matter physics of the ground beneath our feet. In particular, we identify four major challenges concerned with the dynamics in and of the ground: (I) modeling from the grain scale, (II) near-criticality, (III) bridging scales, and (IV) life. For each challenge, we present a selection of topics by individual authors, providing specific context, recent advances, and open questions. Through this, we seek to provide an overview of the opportunities for the broad Soft Matter community to contribute to the fundamental understanding of the physics of the ground, strive towards a common language, and encourage new collaborations across the broad spectrum of scientists interested in the matter of the Earth's surface.