The use of non-orthogonal orbitals allows the construction and use of more compact wave functions than offered by standard methods using orthogonal molecular orbitals; in particular, for molecules containing partly occupied atomic orbitals as present, for example, in transition metal complexes. With the purpose of developing efficient dynamic correlation methods, we discuss several new internal correlation methods employing a reference state containing non-orthogonal active orbitals. The non-orthogonal internally contracted perturbation theory approach is improved in several directions. The major improvements are the use of the Dyall Hamiltonian including two-electron interactions within the active space as the zero-order operator, the calculation of third-order energy-corrections, and the inclusion of excitations in the space of active orbitals. The latter improvement corrects for the use of an incomplete reference state. The improvements are tested for the nitrogen molecule and the challenging chromium dimer. The combined use of the improved zero-order Hamiltonian and the inclusion of active space excitations allow us to obtain potential curves for the chromium dimer that are close to those obtained using the larger complete active space reference wave function.