High herbivore pressure is expected to benefit unpalatable species that co-occur with palatable browsed species. However, for five unpalatable understory species we found no evidence of benefit from deer browse. Detailed studies of one species in natural populations, Arisaema triphyllum, revealed surprising changes in its population structure and demography: deer browse level on a palatable species significantly correlates with reduced plant size and seed rain and male-biased sex ratios of co-occurring Arisaema populations. Analyses of individual size in five unpalatable forest plant species in long-term experimental paired deer exclosure/deer access plots corroborate the natural site results; all five species were smaller in deer access plots. Analyses of abiotic variables in natural and experimental sites suggest one potential mechanism for indirect effects of deer. Deer-mediated soil quality declines included increased soil penetration resistance and decreased leaf litter depth, which are known to hinder plant growth. Our results are likely applicable to other unpalatable forest species and have clear consequences for understory biodiversity. Unpalatable plant species in forests experiencing high deer numbers may be in decline along with their palatable neighbors. Our study implicates deer overabundance in the cascade of forest species decline and the urgency of this conservation issue in North America.