Technological advancements in the molecular characterization of cancers have enabled researchers to identify an increasing number of key molecular drivers of cancer progression. These discoveries have led to multiple novel anticancer therapeutics, and clinical benefit in selected patient populations. Despite this, the identification of clinically relevant predictive biomarkers of response continues to lag behind. In this review, we discuss strategies for the molecular characterization of cancers and the importance of biomarkers for the development of novel antitumor therapeutics. We also review critical successes and failures in oncology, and detail the lessons learnt, which may aid in the acceleration of anticancer drug development and biomarker discovery.