CaCo2O4/rGO was prepared by combining a sol-gel strategy and mechanical ball milling method. The Rietveld refinement results demonstrated a single-phase structure with a monoclinic symmetry. When utilized as an anode for lithium-ion batteries, it exhibited excellent rate performance and electrochemical stability due to the significantly decreasing particle size as well as the formation of a conductive rGO network in the composite after ball milling. A reversible stable specific capacity of 778 mA h g-1 was attained at a current density of 200 mA g-1 after 100 cycles in the range of 0.01-3 V (vs. Li/Li+). The long cycling capacity of 608.5 mA h g-1 persisted even after 300 cycles at 1 A g-1. Thus, the evolution of the CaCo2O4/rGO structures and phases during electrochemical cycling processes was thoroughly investigated for the first time using in situ analytical techniques to elucidate the Li+ storage mechanism. Furthermore, the excellent electrochemical performance of CaCo2O4/rGO as an anode material for sodium-ion batteries with a reversible specific capacity of 400 mA h g-1 that was achieved at a current density of 20 mA g-1 between 0.01 and 3 V (vs. Na/Na+) was also reported for the first time.