We present a general adaptive latent space tuning approach for improving the robustness of machine learning tools with respect to time variation and distribution shift. We demonstrate our approach by developing an encoder-decoder convolutional neural network-based virtual 6D phase space diagnostic of charged particle beams in the HiRES ultrafast electron diffraction (UED) compact particle accelerator with uncertainty quantification. Our method utilizes model-independent adaptive feedback to tune a low-dimensional 2D latent space representation of ∼1 million dimensional objects which are the 15 unique 2D projections (x,y),...,(z,p_{z}) of the 6D phase space (x,y,z,p_{x},p_{y},p_{z}) of the charged particle beams. We demonstrate our method with numerical studies of short electron bunches utilizing experimentally measured UED input beam distributions.