We report structural transformation of sixfold vortex domains into two-, four-, and eightfold vortices via a different type of topological defect in hexagonal manganites. Combining high-resolution electron microscopy and Landau-theory-based numerical simulations, we investigate the remarkable atomic arrangement and the intertwined relationship between the vortex structures and the topological defects. The roles of their displacement field, formation temperature, and nucleation sites are revealed. All conceivable vortices in the system are topologically classified using homotopy group theory, and their origins are identified.