A temperature difference between two electrolyte-immersed electrodes often yields a voltage Δψ between them. This electrolyte Seebeck effect is usually explained by cations and anions flowing differently in thermal gradients. However, using molecular simulations, we found almost the same Δψ for cells filled with pure water as with aqueous alkali halides. Water layering and orientation near polarizable electrodes cause a large temperature-dependent potential drop χ there. The difference in χ of hot and cold electrodes captures most of the thermovoltage, Δψ≈χ_{hot}-χ_{cold}.