We find that the vibrational potential of the O-H stretch vibrations of liquid water shows extreme anharmonicity that arises from the O-H O hydrogen bond interaction. We observe that already in the second excited state of the O-H stretch vibration, the hydrogen atom becomes delocalized between the oxygen atoms of two neighboring water molecules. The energy required for this delocalization is unexpectedly low and corresponds to less than 20% of the dissociation energy of the O-H bond of the water molecule in the gas phase.