This paper presents a neuromorphic processing system with a spike-driven spiking neural network (SNN) processor design for always-on wearable electrocardiogram (ECG) classification. In the proposed system, the ECG signal is captured by level crossing (LC) sampling, achieving native temporal coding with single-bit data representation, which is directly fed into an SNN in an event-driven manner. A hardware-aware spatio-temporal backpropagation (STBP) is suggested as the training scheme to adapt to the LC-based data representation and to generate lightweight SNN models. Such a training scheme diminishes the firing rate of the network with little plenty of classification accuracy loss, thus reducing the switching activity of the circuits for low-power operation. A specialized SNN processor is designed with the spike-driven processing flow and hierarchical memory access scheme. Validated with field programmable gate arrays (FPGA) and evaluated in 40 nm CMOS technology for application-specific integrated circuit (ASIC) design, the SNN processor can achieve 98.22% classification accuracy on the MIT-BIH database for 5-category classification, with an energy efficiency of 0.75 μJ/classification.