Two electron spins in quantum dots coupled through coherent tunneling are generally acknowledged to approximately obey Heisenberg isotropic exchange. This has not been established for two holes. Here we measure the spectra of two holes and of two electrons in two vertically stacked self-assembled InAs quantum dots using optical spectroscopy as a function of electric and magnetic fields. We find that the exchange is approximately isotropic for both systems, but that significant asymmetric contributions, arising from spin-orbit and Zeeman interactions combined with spatial asymmetries, are required to explain large anticrossings and fine-structure energy splittings in the spectra. Asymmetric contributions to the isotropic Hamiltonian for electrons are of the order of a few percent while those for holes are an order of magnitude larger.