Ancestral sequence reconstruction (ASR) is typically performed using homogeneous evolutionary models, which assume that the same substitution propensities affect all sites and lineages. These assumptions are routinely violated: heterogeneous structural and functional constraints favor different amino acid states at different sites, and these constraints often change among lineages as epistatic substitutions accrue at other sites. To evaluate how realistic violations of the homogeneity assumption affect ASR, we developed site-specific substitution models and parameterized them using data from deep mutational scanning experiments on three protein families; we then used these models to perform ASR on the empirical alignments and on alignments simulated under heterogeneous conditions derived from the experiments. Extensive among-site and -lineage heterogeneity is present in these datasets, but the sequences reconstructed from empirical alignments are almost identical, irrespective of whether heterogeneous or homogeneous models are used for ASR. The rare differences occur primarily when phylogenetic signal is weak - at fast-evolving sites and nodes connected by long branches. When ASR is performed on simulated data, errors in the reconstructed sequences become more likely as branch lengths increase, but incorporating heterogeneity into the model does not improve accuracy. These data establish that ASR is robust to unincorporated realistic forms of evolutionary heterogeneity, because the primary determinant of ASR is phylogenetic signal, not the substitution model. The best way to improve accuracy is therefore not to develop more elaborate models but to apply ASR to densely sampled alignments that maximize phylogenetic signal at the nodes of interest.