Aligning a pair of images in a mid-space is a common approach to ensuring that deformable image registration is symmetric - that it does not depend on the arbitrary ordering of the input images. The results are, however, generally dependent on the choice of the mid-space. In particular, the set of possible solutions is typically affected by the constraints that are enforced on the two transformations (that deform the two images), which are to prevent the mid-space from drifting too far from the native image spaces. The use of an implicit atlas has been proposed to define the mid-space for pairwise registration. In this work, we show that by aligning the atlas to each image in the native image space, implicit-atlas-based pairwise registration can be made independent of the mid-space, thereby eliminating the need for anti-drift constraints. We derive a new symmetric cost function that only depends on a single transformation morphing one image to the other, and validate it through diffeomorphic registration experiments on brain magnetic resonance images.