The presence of a weak second-harmonic field in an intense-laser ionization experiment affects the momentum-resolved electron yield, depending on the relative phase between the ω and the 2ω component. The proposed two-color "phase-of-the-phase spectroscopy" quantifies for each final electron momentum a relative-phase contrast (RPC) and a phase of the phase (PP) describing how much and with which phase lag, respectively, the yield changes as a function of the relative phase. Experimental results for RPC and PP spectra for rare gas atoms and CO_{2} are presented. The spectra demonstrate a rather universal structure that is analyzed with the help of a simple model based on electron trajectories, wave-packet spreading, and (multiple) rescattering. Details in the PP and RPC spectra are target sensitive and, thus, may be used to extract structural (or even dynamical) information with high accuracy.