Commercially available, synthesis-on-demand virtual libraries contain upward of trillions of readily synthesizable compounds for drug discovery campaigns. These libraries are a critical resource for rapid cycles of in silico discovery, property optimization and in vitro validation. However, as these libraries continue to grow exponentially in size, traditional search strategies encounter significant limitations. Here we present NeuralGenThesis (NGT), an efficient reinforcement learning approach to generate compounds from ultralarge libraries that satisfy user-specified constraints. Our method first trains a generative model over a virtual library and subsequently trains a normalizing flow to learn a distribution over latent space that decodes constraint-satisfying compounds. NGT allows multiple constraints simultaneously without dictating how molecular properties are calculated. Using NGT, we generated potent and selective inhibitors for the melanocortin-2 receptor (MC2R) from a three trillion compound library. NGT offers a powerful and scalable solution for navigating ultralarge virtual libraries, accelerating drug discovery efforts.