A Deep Learning-based Pipeline for Segmenting the Cerebral Cortex Laminar Structure in Histology Images

Characterizing the anatomical structure and connectivity between cortical regions is a critical step towards understanding the information processing properties of the brain and will help provide insight into the nature of neurological disorders. A key feature of the mammalian cerebral cortex is its laminar structure. Identifying these layers in neuroimaging data is important for understanding their global structure and to help understand the connectivity patterns of neurons in the brain. We studied Nissl-stained and myelin-stained slice images of the brain of the common marmoset (Callithrix jacchus), which is a new world monkey that is becoming increasingly popular in the neuroscience community as an object of study. We present a novel computational framework that first acquired the cortical labels using AI-based tools followed by a trained deep learning model to segment cerebral cortical layers. We obtained a Euclidean distance of $1274.750\pm 156.400$ $\mu m$ for the cortical labels acquisition, which was in the acceptable range by computing the half Euclidean distance of the average cortex thickness ( $1800.630\mu m$ ). We compared our cortical layer segmentation pipeline with the pipeline proposed by Wagstyl et al. (PLoS biology, 18(4), e3000678 2020) adapted to 2D data. We obtained a better mean ${95}^{\mathrm{th}}$ percentile Hausdorff distance (95HD) of $92.150\mu m$ . Whereas a mean 95HD of $94.170\mu m$ was obtained from Wagstyl et al. We also compared our pipeline's performance against theirs using their dataset (the BigBrain dataset). The results also showed better segmentation quality, $85.318\%$ Jaccard Index acquired from our pipeline, while $83.000\%$ was stated in their paper.