Artificial neural networks have recently gained significant attention in biomedical research. However, their utility in survival analysis still faces many challenges. In addition to designing models for high accuracy, it is essential to optimize models that provide biologically meaningful insights. With these considerations in mind, we developed a deep neural network model, MaskedNet, to identify genes and pathways whose expression at the time of diagnosis is associated with overall survival. MaskedNet was trained using TCGA breast cancer transcriptome and clinical data, and the model's final output was the predicted logarithm of the hazard ratio for death. The trained model was interpreted using SHapley Additive exPlanations (SHAP), a technique grounded in robust mathematical principles that assigns importance scores to input features. Compared to traditional Cox proportional hazards regression, MaskedNet had higher accuracy, as measured by Harrell's C-index. We also found that aggregating outputs from several model runs identified multiple genes and pathways associated with overall survival, including IFNG and PIK3CA genes, along with their related pathways. To further elucidate the role of the IFNG gene, tumors were partitioned into two groups based on low and high IFNG SHAP values, respectively. Tumors with lower IFNG SHAP values exhibited higher IFNG expression and better overall survival, which were linked to more abundant presence of M1 macrophages and activated CD4+ and CD8+ T cells in the tumor microenvironment. The association of the IFNG pathway with overall survival was validated in the trastuzumab arm of the NCCTG-N9831 trial, an independent breast cancer study.