Whether mammal-microbiome interactions are persistent and specific over evolutionary time is controversial. Here we show that host phylogeny and major dietary shifts have affected the distribution of different gut bacterial lineages and did so on vastly different bacterial phylogenetic resolutions. Diet mostly influences the acquisition of ancient and large microbial lineages. Conversely, correlation with host phylogeny is mostly seen among more recently diverged bacterial lineages, consistent with processes operating at similar timescales to host evolution. Considering microbiomes at appropriate phylogenetic scales allows us to model their evolution along the mammalian tree and to infer ancient diets from the predicted microbiomes of mammalian ancestors. Phylogenetic analyses support co-speciation as having a significant role in the evolution of mammalian gut microbiome compositions. Highly co-speciating bacterial genera are also associated with immune diseases in humans, laying a path for future studies that probe these co-speciating bacteria for signs of co-evolution.