Pharmacovigilance seeks to detect and describe adverse drug reactions early. Ideally, we would like to see objective evidence that a chosen signal detection approach can be expected to be effective. The development and evaluation of evidence-based methods require benchmarks for signal detection performance, and recent years have seen unprecedented efforts to build such reference sets. Here, we argue that evaluation should be made against emerging and not established adverse drug reactions, and we present real-world examples that illustrate the relevance of this to pharmacovigilance methods development for both individual case reports and longitudinal health records. The establishment of broader reference sets of emerging safety signals must be made a top priority to achieve more effective pharmacovigilance methods development and evaluation.