Automatic detection of major depressive disorder (MDD) with multiple-channel electroencephalography (EEG) signals is of great significance for treatment of the mental diseases. In a U-net network, clear EEG signals are fed to obtain temporal feature tensor through encoder and decoder networks with several convolution operations. Moreover, the clear EEG signals can be converted into multi-scale spectrogram to obtain the rich saliency information and then the spectrogram feature tensor can be extracted by another symmetrical U-net. The temporal and spectrogram feature tensors can provide more comprehensive information, but may also contain redundant information, which may affect the detection of MDD. To deal with such issue, this paper proposed a novel Temporal Spectrogram Unet (TSUnet-CC), which embeds the cross channel-wise attention mechanism for multiple-channel EEGbased MDD identification. We make three novel contributions: 1) multi-scale saliency-encoded spectrogram using Fourierbased approach to capture rich saliency information under different scales, 2) TSUnet network using a symmetrical twostream U-net architecture that learns multiple temporal and spectrogram feature tensors in time and frequency domains, and 3) cross channel-wise block enabling the larger weights of key feature channels that contain MDD information. The leaveone-subject-out experiments show that our proposed TSUnetCC gains high performance with a classification accuracy up to 98.55% and 99.22% in eyes closed and eyes open datasets, which outperformed some state-of-the-art methods and revealed its clinical potential.