Image-based phenotypic drug profiling is receiving increasing attention in drug discovery and precision medicine. Compared to classical end-point measurements quantifying drug response, image-based profiling enables both the quantification of drug response and characterization of disease entities and drug-induced cell-death phenotypes. Here, we aim to quantify image-based drug responses in patient-derived 3D spheroid tumor cell cultures, tackling the challenges of a lack of single-cell-segmentation methods and limited patient-derived material. Therefore, we investigate deep transfer learning with patient-by-patient fine-tuning for cell-viability quantification. We fine-tune a convolutional neural network (pre-trained on ImageNet) with 210 control images specific to a single training cell line and 54 additional screen -specific assay control images. This method of image-based drug profiling is validated on 6 cell lines with known drug sensitivities, and further tested with primary patient-derived samples in a medium-throughput setting. Network outputs at different drug concentrations are used for drug-sensitivity scoring, and dense-layer activations are used in t-distributed stochastic neighbor embeddings of drugs to visualize groups of drugs with similar cell-death phenotypes. Image-based cell-line experiments show strong correlation to metabolic results ( R ≈ 0.7 ) and confirm expected hits, indicating the predictive power of deep learning to identify drug-hit candidates for individual patients. In patient-derived samples, combining drug sensitivity scoring with phenotypic analysis may provide opportunities for complementary combination treatments. Deep transfer learning with patient-by-patient fine-tuning is a promising, segmentation-free image-analysis approach for precision medicine and drug discovery.