We develop a general theory of Fermi polarons at nonzero temperature, including particle-hole excitations of the Fermi sea shakeup to arbitrarily high orders. The exact set of equations of the spectral function is derived by using both Chevy ansatz and diagrammatic approach, and their equivalence is clarified to hold in free space only, with an unregularized infinitesimal interaction strength. The correction to the polaron spectral function arising from two-particle-hole excitations is explicitly examined for an exemplary case of Fermi polarons in one-dimensional optical lattices. We find quantitative improvements at low temperatures with the inclusion of two-particle-hole excitations, in both polaron energies and decay rates. Our exact theory of Fermi polarons with arbitrary orders of particle-hole excitations might be used to better understand the intriguing polaron dynamical responses in two or three dimensions, whether in free space or within lattices.