Diagnosis of epilepsy based on visual inspection of electroencephalogram (EEG) abnormalities is an inefficient, time-consuming, and expert-centered process. Moreover, the diagnosis based on ictal epileptiform events is challenging as the ictal patterns are infrequent. Consequently, the development of an automated, fast, and reliable epileptic EEG diagnostic system is essential. The interictal epileptiform discharges (IEDs) are recurring patterns that are highly suggestive of epilepsy. In this paper, we propose an epileptic EEG classification system based on IED detection. The proposed system comprises of three modules: pre-processing, waveform-level classification, and EEG-level classification. We employ a Convolutional Neural Network (CNN) for waveform-level classification and a Support Vector Machine (SVM) for EEG-level classification. We evaluated the proposed system on a dataset of 156 EEGs recorded at Massachusetts General Hospital (MGH), Boston. The system achieved a mean 4-fold classification accuracy of 83.86% for classifying EEGs with and without IEDs.