Doxorubicin (DOX) is a widely-used anticancer drug, but its cardiotoxicity severely hampers its potency in chemotherapy. Herein, human serum albumin (HSA) is engaged as a biocompatible nanocarrier to load a pH-sensitive DOX prodrug, DMDOX, generating HSA-DMDOX nanoparticles via self-assembly driven by hydrophobic interactions. HSA-DMDOX disperses well in a physiological environment (∼40 nm) but aggregates in a tumor acidic microenvironment (pH 6.5, ∼140 nm) owing to the hydrophobicity increase of DMDOX by protonation of carboxylic groups. In vitro anticancer study showed that HSA-DMDOX exhibited enhanced cellular uptake by 4T1 cells and superior cytotoxicity in comparison to HSA-DOX nanoparticles. In vivo study suggested that HSA-DMDOX achieved long blood circulation, aggregation enhanced tumor retention, comparable antitumor efficacy and reduced cardiotoxicity relative to free DOX. Our work presents a facile and effective approach to delivering anthracyclines by HSA-based tumor pH-responsive nanoparticles with aggregation-enhanced tumor retention and reduced toxicity.