Theoretical calculations show that twisted double bilayer graphene (TDBG) under a transverse electric field develops a valley Chern number 2 at charge neutrality. Using thermodynamic and thermal activation measurements we report the experimental observation of a universal closing of the charge neutrality gap in the Hofstadter spectrum of TDBG at 1/2 magnetic flux per unit cell, in agreement with theoretical predictions for a valley Chern number 2 gap. Our theoretical analysis of the experimental data shows that the interaction energy, while larger than the flat-band bandwidth in TDBG near 1° does not alter the emergent valley symmetry or the single-particle band topology.