As an alternative to target-driven drug discovery, phenotype-driven approaches identify compounds that counteract the overall disease effects by analyzing phenotypic signatures. Our study introduces a novel approach to this field, aiming to expand the search space for new therapeutic agents. We introduce PDGrapher, a causally-inspired graph neural network (GNN) designed to predict combinatorial perturbagens - sets of therapeutic targets - capable of reversing disease effects. Unlike methods that learn responses to perturbations, PDGrapher solves the inverse problem, which is to infer the perturbagens necessary to achieve a specific response - i.e., directly predicting perturbagens by learning which perturbations elicit a desired response. By encoding gene regulatory networks or protein-protein interactions, PDGrapher can predict unseen chemical or genetic perturbagens, aiding in the discovery of novel drugs or therapeutic targets. Experiments across nine cell lines with chemical perturbations show that PDGrapher successfully predicted effective perturbagens in up to 13.33% additional test samples and ranked therapeutic targets up to 35% higher than the competing methods, and the method shows competitive performance across ten genetic perturbation datasets. A key innovation of PDGrapher is its direct prediction capability, which contrasts with the indirect, computationally intensive models traditionally used in phenotype-driven drug discovery that only predict changes in phenotypes due to perturbations. The direct approach enables PDGrapher to train up to 25 times faster than methods like scGEN and CellOT, representing a considerable leap in efficiency. Our results suggest that PDGrapher can advance phenotype-driven drug discovery, offering a fast and comprehensive approach to identifying therapeutically useful perturbations.