If a quantum dot is coupled to a topological superconductor via tunneling contacts, each contact hosts a Majorana zero mode in the limit of zero transmission. Close to a resonance and at a finite contact transparency, the resonant level in the quantum dot couples the Majorana modes, but a ground-state degeneracy per fermion parity subspace remains if the number of Majorana modes coupled to the dot is five or larger. Upon varying shape-defining gate voltages while remaining close to resonance, a nontrivial evolution within the degenerate ground-state manifold is achieved. We characterize the corresponding non-Abelian holonomy for a quantum dot with chaotic classical dynamics using random matrix theory and discuss measurable signatures of the non-Abelian time evolution.