We report the effects of nitrogen diffusion on exchange bias in MnN/CoFeB heterostructures as a function of MnN thickness and field-annealing temperature. We find that competing effects occur in which high-temperature annealing enhances exchange bias in heterostructures with thick MnN through improved crystallinity, but in thinner samples this annealing ultimately eliminates the exchange bias due to nitrogen deficiency. Using polarized neutron reflectometry and magnetic x-ray spectroscopy, we directly observe increasing amounts of nitrogen migration from MnN into the underlying Ta seed layer with increased annealing temperature. In heterostructures with thin MnN layers, the resulting nitrogen deficiency becomes significant enough to alter the antiferromagnetic state before the Ta seed layer is nitrogen saturated. Furthermore, we observe intermixing at the MnN/CoFeB interface which is attributed to the nitrogen deficiency creating vacancies in the MnN layer after annealing in a field. This intermixing of Mn with Co and Fe is not believed to be the cause for loss of exchange bias when the MnN layer is too thin but is instead a secondary effect due to increased vacancies after nitrogen migration.