The optical extinction response of individual Au-Ag@SiO2 heterodimers whose individual morphologies are determined by transmission electron microscopy (TEM) is investigated using spatial modulation spectroscopy. The extinction spectra show two resonances spectrally close to the surface plasmon resonances of the constituting Au and Ag@SiO2 core-shell particles. The interparticle electromagnetic coupling is demonstrated to induce a large increase of the optical extinction of the dimer around its Au-like surface plasmon resonance for light polarized along its axis, as compared to that for perpendicular polarization and to that of an isolated Au nanoparticle. For spherical particles, this interaction also leads to comparable shifts with light polarization of the two dimer resonances, an effect masked or even reversed for particles significantly deviating from sphericity. Both amplitude and spectral effects are found to be in excellent quantitative agreement with numerical simulations when using the TEM-measured dimer morphology (i.e., size, shape, and orientation of the individual dimers), stressing the importance of individual morphology characterization for interpreting heterodimer optical response.