Over and under-sedation are common in critically ill patients admitted to the Intensive Care Unit. Clinical assessments provide limited time resolution and are based on behavior rather than the brain itself. Existing brain monitors have been developed primarily for non-ICU settings. Here, we use a clinical dataset from 154 ICU patients in whom the Richmond Agitation-Sedation Score is assessed about every 2 hours. We develop a recurrent neural network (RNN) model to discriminate between deep vs. no sedation, trained end-to-end from raw EEG spectrograms without any feature extraction. We obtain an average area under the ROC of 0.8 on 10-fold cross validation across patients. Our RNN is able to provide reliable estimates of sedation levels consistently better compared to a feed-forward model with simple smoothing. Decomposing the prediction error in terms of sedatives reveals that patient-specific calibration for sedatives is expected to further improve sedation monitoring.