Accurate and robust classification of Motor Imagery (MI) from Electroencephalography (EEG) signals is among the most challenging tasks in Brain-Computer Interface (BCI) field. To address this challenge, this paper proposes a novel, neuro-physiologically inspired convolutional neural network (CNN) named Filter-Bank Convolutional Network (FBCNet) for MI classification. Capturing neurophysiological signatures of MI, FBCNet first creates a multi-view representation of the data by bandpass-filtering the EEG into multiple frequency bands. Next, spatially discriminative patterns for each view are learned using a CNN layer. Finally, the temporal information is aggregated using a new variance layer and a fully connected layer classifies the resultant features into MI classes. We evaluate the performance of FBCNet on a publicly available dataset from Korea University for classification of left vs right hand MI in a subject-specific 10-fold cross-validation setting. Results show that FBCNet achieves more than 6.7% higher accuracy compared to other state-of-the-art deep learning architectures while requiring less than 1% of the learning parameters. We explain the higher classification accuracy achieved by FBCNet using feature visualization where we show the superiority of FBCNet in learning interpretable and highly generalizable discriminative features. We provide the source code of FBCNet for reproducibility of results.