Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder

An open-source LlaMA2-7b-Cantonese model (https://huggingface.co/indiejoseph) is applied to extract context features for each Cantonese word appearing in the movie. The original LlaMA2-7b, released by Meta [18], has $32$ layers and is trained on a mix of publicly available online data (with English accounting for $89.7\%$ and Chinese accounting for $0.13\%$), with a context length of $4k$. Since it employs a byte pair encoding algorithm to decompose unknown UTF-8 characters, it can also encode the context knowledge of Cantonese. However, to better adapt to the Cantonese context, the LlaMA2-7b-Cantonese used in this study is further trained on the LlaMA2-7b model using additional Cantonese corpus.

The context feature of the word $s_{i}$ is extracted based on the next-word-prediction task: for each word-time pair $\left(s_{i},t_{i}\right)$, LlaMA2-Cantonese takes a word sequence $S=\left(s_{i-255},\cdots,s_{i-1},s_{i}\right)$ as input, and its hidden layer activations provide vector embeddings that represent the meaning of $s_{i}$ within a context length of $256$. This yields a high-dimensional vector-time pair $\left(\bm{X}_{i},t_{i}\right)$ where $\bm{X}_{i}$ is a $4096$-dimensional context representation for $s_{i}$. Then, three steps are conducted before we obtain the final stimulus matrix (as shown in Figure 2): (1) these vectors are resampled at times corresponding to the fMRI acquisitions using a three-lobe Lanczos filter [13, 24]; (2) vectors are reduced to $d=90$ dimensions using PCA for computational efficiency; (3) finally, based on the linearized finite impulse response (FIR) model [25], $\left(\bm{X}_{i-6},\bm{X}_{i-5},\cdots,\bm{X}_{i-1}\right)$, i.e. context representations from $0.9s$ to $5.4s$ earlier before the timepoint $t_{i}$, are concatenated to predict the fMRI signal, $\left(y_{i},t_{i}\right)$. We then obtain the final context stimulus matrix $\bm{X}\in\mathbb{R}^{\#TR\times 6d}$, time-aligned with the processed fMRI signal $\bm{Y}\in\mathbb{R}^{\#TR\times 1}$ of a brain voxel.

Let $f\left(\bm{X}\right)$ represent the brain encoding model. Following the work of [14], we select $f$ as a voxel-wise linear transformation between $\bm{X}$ and $\bm{Y}$ for interpretability. For each subject $s$, voxel $v$, and LlaMA2-Cantonese layer $l_{i}$, we fit a separate encoding model $f_{s,v}\left(\bm{X}_{l_{i}}\right):=\bm{X}_{l_{i}}\bm{W}_{s,v}^{l_{i}}$, using linearized ridge regression, to predict the fMRI signal $\bm{Y}_{s,v}$, where $\bm{W}_{s,v}^{l_{i}}$ are the learnable weight parameters. The final objective function is:

where $\|\cdot\|_{F}$ denotes the Frobenius norm, and $\lambda_{s,v}$ is the only hyperparameter representing the regularization weight. Each $\lambda_{s,v}$ is selected independently for each voxel in each subject.

Specifically, $20\%$ of the data samples (continuous in the time dimension) are held out as the test set. In contrast, the remaining data samples are divided into training and validation sets for regression and hyperparameter selection using a bootstrap method [13]. In each iteration, the model weights are estimated on the training set for each of $10$ possible regularization coefficients (log spaced between $10$ and $1000$). These weights are used to predict responses in the validation set, followed by the calculation of $R^{2}$ between the actual and predicted fMRI time series. The regularization coefficient for each voxel is chosen based on the value that yields the best performance on the validation set, averaged across 50 bootstraps. After all parameters are finalized, the encoding model is applied to the test set to obtain the brain score (i.e., $R^{2}$) for each voxel.

Brain score analysis of the whole brain. We computed the brain score for the entire brain of each subject, averaging across all voxels, with various activation layers of LlaMA2-Cantonese. The Pearson correlation between the brain score and the subject group’s MoCA score was subsequently calculated using pearsonr in Scipy toolkit of Python.

Brain score analysis within language-related ROIs. We focus on language-related ROIs for two reasons: 1) early functional changes in NCD subjects occur in language brain areas; 2) the movie-watching task could activate brain regions related to vision, and the ROI analysis can reduce the influence from other cognitive processes to some extent.

Consequently, our study uses the brain parcellation from the Destrieux cortical deterministic atlas (dated $2009$) [26] to identify language-related ROIs [27, 28] in both brain hemispheres, including the precuneus, angular gyrus (AG), inferior temporal gyrus (ITG), middle temporal gyrus (MTG), superior temporal gyrus (STG), superior frontal gyrus (SFG), middle frontal gyrus (MFG), and inferior frontal gyrus (IFG). These ROIs correspond to a total of 26 labels in the Destrieux atlas. We then analyzed the brain scores of each ROI and their Pearson correlations with MoCA scores.

Statistical significance To assess the statistical significance of the Pearson correlations between brain scores and MoCA scores, we conducted two-tailed t-tests with $p<0.05$. The Benjamini-Hochberg False Discovery Rate (FDR) [29] correction was used to adjust the p-values for multiple comparisons.

The blue bars in Figure 3 display the average brain scores for the entire subject group. The $95$ subjects are further divided into two subgroups: the higher cognitive subgroup with MoCA scores $>20$ and the lower cognitive subgroup with MoCA $\leq 20$, where $20$ is the median of their MoCA scores.

The overall trend indicates that the average brain scores for different cognitive groups reach peaks simultaneously in the relatively early layers (from the 7th to the 16th layer), followed by a gradual decline. Their trend lines align well with [14], which reported that the performances of semantic encoding models for three young adults fitted by LlaMA2-30B or LlaMA2-65B reach a peak between $20\%$ and $40\%$ of the layer depth. These similar patterns across different cognitive subgroups and language stimuli suggest both the effectiveness and the general nature of the encoding model fitted by LlaMA2.

Comparing the two cognitive subgroups, the average brain score for the higher cognitive group is consistently better than that of the lower cognitive group. The averaged $\Delta(R^{2})=$1.28\text{\times}{10}^{-3}$$ across $32$ layers, with a standard deviation of $1.53\text{\times}{10}^{-4}$ indicating that the fMRI signal within the higher cognitive subgroup more closely aligns with context representations. This gap could arise from two possible reasons: 1) individuals with higher cognitive abilities may have followed the dialogue more closely during the task; or 2) their brains could process context information more similarly to LlaMA2.

The Pearson correlation between the brain scores and MoCA scores for the entire group is depicted as the blue line in Figure 3. Correlation coefficients reach $0.25$ in layers $1$, $2$, and $8$, and drop significantly after layer $16$. Along with brain score performance, we focus on the encoding model fitted by the 8th layer in the subsequent ROI analysis.

In Figure 4, the average brain score and correlation coefficient of each ROI are projected onto the fsaverage5 brain surface using vol to surf in nilearn toolkit. As illustrated in Figure 4c, only $9$ out of $26$ ROIs pass the significance test for correlation ($p$ $<0.05$ adjusted using FDR): MTG (labels $38$, $113$), Precuneus (labels $30$, $105$), AG (labels $25$, $100$), Left-SFG (label $16$), and SFS (labels $55$, $130$).

The lateral STG (labels $109$, $34$), AG, and the plan-tempo STG in the left hemisphere (label $36$) exhibit higher brain scores (figure 4a). However, only the AG is significantly correlated with the MoCA score, with $r_{100}=0.253$ and $r_{25}=0.245$ (figure 4b,c). We propose that the STG primarily encodes lower-level semantic information, as supported by [30, 31]. Consequently, the STG may be associated with perception but not strongly correlated with high-level semantic understanding.

To our surprise, although the MTG in the left hemisphere has a medium brain score of $8.7\text{\times}{10}^{-3}$, it attains the highest correlation coefficient of $r_{38}=0.37$ (adjusted $p<0.001$). The detailed regression between the brain score of MTG and the MoCA score is plotted in Figure 4d. These findings are consistent with [16], which reported that the correlation between brain scores and comprehension scores peaks in the AG and MTG.

To investigate the impact of additional Cantonese corpus on LLaMa2-Cantonese, we refitted the encoding model using the representations of the original LLaMa2 and calculated the average brain scores along with their correlations to MoCA scores. Results in Table 2 indicate that no significant difference in brain scores or correlations exist, except for the correlation at layer $8$, which is $4.9\%$ higher after Cantonese pretraining. This can be attributed to the fact that Cantonese pretraining did not alter the structure or embedding representations of LLaMA2.