Online classification of imagined speech using functional near-infrared spectroscopy signals

Twelve able-bodied participants (seven males) between the ages of 23 and 33 (mean age: 28.4  ±  2.9 years) participated in this study. Participants were fluent in English, had normal or corrected-to-normal vision, and had no health issues that could adversely affect the measurements or their ability to follow the experimental protocol. These issues included neurological, cardiovascular, respiratory, psychiatric, metabolic, degenerative, or alcohol-related conditions. Participants were asked to refrain from drinking alcoholic or caffeinated beverages at least 3 h prior to each session. This study was approved by the research ethics boards of the Holland Bloorview Kids Rehabilitation Hospital and the University of Toronto. Written consent was obtained from all participants prior to study participation.

fNIRS measurements were collected from the frontal, parietal and temporal cortices using a continuous-wave near-infrared spectrometer (ETG-4000 Optical Topography System, Hitachi Medical Co., Japan). As shown in figure 1, 16 NIR emitters and 14 photodetectors were integrated in two 3  ×  5 rectangular grids of optical fibers in a standard EEG cap (EasyCap, Germany). Each NIR emitter contained two laser diodes that simultaneously emitted NIR light at wavelengths of 695 nm and 830 nm. The optical signals were sampled at 10 Hz.

Zoom In Zoom Out Reset image size

Adjacent positions in each of the two 3  ×  5 grids, were 3 cm apart. Only optical signals arising from source-detector pairs (or 'channels') separated by 3 cm were acquired for analysis. This separation distance yielded a depth penetration of light between 2 and 3 cm [29, 30], which surpasses the average scalp-to-cortex depth within the brain areas monitored [31]. Using this configuration, optical signals were acquired from a total of 44 measurement sites on the cerebral cortex, 22 on each hemisphere (see figure 1). In addition to fNIRS measurements, EEG signals were recorded from 32 locations using BrainAmp DC amplifier (Brain Products GmbH, Germany). These data are not analyzed herein.

Participants attended two sessions on two separate days, within a span of 6–21 d. The first session consisted of three blocks, starting with an offline block and followed by two online blocks. In the offline block, participants performed 36 trials, including 12 'yes' imagined speech trials, 12 'no' imagined speech trials and 12 unconstrained rest trials. The trials were presented in a pseudorandom order. At the end of the offline block, a 3-class classifier was trained using the data from the offline block. Each online block consisted of 24 trials, eight trials per class, presented in a pseudorandom order. Participants were presented with the classifier decision subsequent to each trial. The 3-class classifier was re-trained after each block using the data from all previous blocks.

The second session consisted of four online blocks, each with 24 trials equally distributed among the three classes presented in pseudorandom order. Similar to the first session, the 3-class classifier was retrained after each block. The timing diagram is depicted in figure 2.

Zoom In Zoom Out Reset image size

A fixation cross appeared at the center of a blank screen at the beginning of each trial and persisted throughout the trial. Each trial started with a 14 s baseline period which allowed the hemodynamic signal to return to a basal level [32, 33]. Participants were asked to refrain from performing any of the imagined speech tasks during this period. They had no knowledge of the type of the next trial at the time of baseline collection.

In the imagined speech trials, a question appeared on the screen (below the fixation cross) after the baseline period for 3 s. Then it was replaced by the instruction 'start', which disappeared after 1 s. The question was always the same: 'Is this word in uppercase letters? WORD'. For the yes trials, the word was written in uppercase letters. For the no trials, the word was written in lowercase letters. The words were different in each question and were selected at random from a list of emotionally neutral words suggested by [34]. In the unconstrained rest trials, the phrase 'rest' appeared on the screen (again, below the fixation cross) for 3 s, which was then replaced by the instruction, 'start', for 1 s.

Participants were instructed to commence the mental task as soon as the 'start' instruction disappeared. For the imagined speech trials, participants were instructed to think 'yes' or 'no' while iteratively repeating the word 'yes' or 'no' mentally. They were explicitly instructed to perform the task without any vocalization or motor movement, especially of the lips, tongue or jaw. In the unconstrained 'rest' trials, participants allowed normal thought processes to occur without restriction. The participant was asked to perform the mental task for 15 s for all trial types. This duration was determined based on previous similar fNIRS studies and the suggested minimum measurement time for a hemodynamic response in literature [21].

At the end of each session, the participants were asked to rate from 1 to 5 (where 1 was the lowest and 5 the highest) their perceived ability to perform the task (data not shown).

2.4.1. Signal processing.

2.4.2. Baseline removal.

2.4.3. Feature extraction.

2.4.4. Classification.

For classification, a regularized linear discriminant analysis (RLDA) algorithm was used [42]. This method was chosen as it led to the highest average accuracy during the pilot sessions compared to support vector machines (linear, polynomial, radial basis function and sigmoid kernels), neural networks (multilayer perceptron with one hidden layer) and naïve Bayes classifiers.

To discriminate between the three classes, a multiclass LDA was used for classification. In contrast with other types of discriminant analysis, e.g. quadratic discriminant analysis, LDA assumes that all classes have the same covariance. This common pooled covariance matrix is defined as:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle \widehat{\Sigma }=\sum\limits_{k=1}^{K}{\sum\limits_{i\in {{I}_{k}}}{\left( {{X}_{i}}-{{\mu }_{k}} \right){{\left( {{X}_{i}}-{{\mu }_{k}} \right)}^{T}}/(N-K)}}\nonumber \end{align} \tag{ 1 }$$

where $K$ is the number of classes, ${{X}_{i}}$ is the feature vector for the $i{\rm th}$ example, ${{I}_{k}}=\{i~|~{{y}_{i}}=k\}$ is the subset of indices identifying the examples of the $k{\rm th}$ class, ${{y}_{i}}$ is the class label of the $i{\rm th}$ example, ${{\mu }_{k}}$ is the mean of all examples of the $k{\rm th}$ class, and $N$ is the total number of examples.

LDA classification is done based on the analysis of the following two scatter matrices: the within-class scatter matrix and the between-class scatter matrix. The within-class scatter matrix can be expressed in terms of the common covariance matrix defined in equation (1):

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{S}_{{\rm w}}}=\left( N-K \right)\times \widehat{\Sigma }.\nonumber \end{align} \tag{ 2 }$$

The between-class scatter matrix is defined as:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{S}_{{\rm b}}}=\underset{k=1}{\overset{K}{\mathop \sum }}\,{{N}_{k}}({{\mu }_{k}}-\mu){{({{\mu }_{k}}-\mu)}^{T}}\nonumber \end{align} \tag{ 3 }$$

where $\mu$ is the overall mean of all examples and ${{N}_{k}}$ is the number of examples in the $k{\rm th}$ class, or ${{N}_{k}}=\left| {{I}_{k}} \right|$ where $\left| . \right|$ denotes cardinality.

The main goal of LDA is to find a set of coefficients, ${\rm W}$, that maximizes the following ratio:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{W}_{{\rm LDA}}}=\underset{W}{\mathop{{\rm argmax}}}\,\frac{{{W}^{T}}{{S}_{{\rm b}}}W}{{{W}^{T}}{{S}_{{\rm w}}}W}.\nonumber \end{align} \tag{ 4 }$$

This ratio is called the Fisher criterion.

In regularized LDA, the common pooled covariance matrix is replaced with the following covariance matrix for each class:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{\widehat{\Sigma }}_{\gamma }}=\left( 1-\gamma \right)\widehat{\Sigma }+\gamma \cdot {\rm diag}\left( \widehat{\Sigma } \right)\nonumber \end{align} \tag{ 5 }$$

where ${\rm diag}\left( \widehat{\Sigma } \right)$ are the diagonal elements of $\widehat{\Sigma }$ and $\gamma$ is the regularization parameter. It can be seen that when $\gamma$ is equal to zero, ${{\widehat{\Sigma }}_{\gamma }}$ is equal to $\widehat{\Sigma }$, and the optimization equation reduces to that of non-regularized LDA.

2.4.5. Optimization of the regularization parameter.

The only hyper-parameter in an RLDA classifier is the regularization parameter, gamma, which can be any value between 0 and 1. This parameter was optimized every time the classifier was trained using a leave-one-out cross-validation (LOOCV) method. Prior to each online block, we calculated the LOOCV accuracy on the data from all previous blocks for different gamma values in the range of 0–1 in 0.05 increments, with two exceptions which are explained below. The gamma which resulted in the highest LOOCV accuracy was selected for subsequent classifier training. In case of a tie, the largest gamma was selected to obtain a more generalized classifier. The classifier was then trained on the entire training set using that gamma.

Two restrictions were applied to the gamma range. Firstly, during each session, the maximum value of gamma in the gamma-optimization step was set to the gamma used in the previous block. Since more same-day data was acquired as the session progressed, the need for generalizing the classifier was reduced and the classifier could be further optimized. Secondly, at the beginning of the second session, a minimum of 0.3 was used for gamma to prevent overfitting, i.e. over-emphasis on data from the first session and thereby preserve generalizability. The value of 0.3 was optimized and selected during pilot sessions. The data analysis steps are summarized in figure 3.

Zoom In Zoom Out Reset image size

Table 1 provides the online 3-class classification accuracies obtained during the six online blocks performed by each participant. Nine out of twelve participants reached above-chance online classification accuracy in their final three blocks (p  <  0.001 using the binomial cumulative distribution [43]), achieving an average online accuracy above the 70% threshold (minimum acceptable threshold for practical BCI applications [10]).

Excluding three participants, P5, P7 and P11, whose second sessions were interrupted upon participants' request to remove the cap due to discomfort and fatigue.

For participants P5, P7 and P11, the second session was interrupted as these participants asked to have the cap removed due to discomfort. After the removal of the cap, these participants took a short break and continued the experiment. In the post-session questionnaire, in response to the question of 'How hard was it to perform the mental tasks?', all three of these participants chose 5 (on a scale of 1–5, 5 being the hardest) and stated that that the task was difficult (5 on a scale of 1–5) to perform given the discomfort of the cap. Due to this reason, as well as potential variations in fNIRS cap positioning between successive donning of the cap, the mean accuracy is also reported without these three participants in table 1. Note that calculating the mean accuracy without these three participants is done only as a secondary analysis and is not part of the primary results and discussion.

In order to determine the role of each fNIRS channels in providing the discriminative information, we used the value of the Fisher criterion calculated for each feature. Since RLDA was used for classification (which works based on maximizing the Fisher criterion) and each fNIRS channel produced only one feature, the calculated Fisher criterion for that feature represented the level of discriminative information that fNIRS channel provided. Figure 4(a) depicts the brain map of the calculated Fisher criterion for each channel averaged across participants. Figure 4(b) provides the brain map of the standard deviation of the calculated Fisher criterion across participants.

Zoom In Zoom Out Reset image size

In this paper, we proposed an online 3-class BCI based on imagined speech. An average ternary classification accuracy of 71.5%  ±  24.7% was reached across all participants in their last block, with nine out of 12 participants surpassing the chance level (p  <  0.001 using the binomial cumulative distribution [43]).

Some studies have explored the possibility of developing a multiclass (>2 classes) BCI using fNIRS [44]. Power et al [6] developed an fNIRS-BCI to classify between mental singing, mental arithmetic and unconstrained rest and reported an offline ternary classification accuracy of 56.2%  ±  8.7% across seven participants. Herff et al [45] used fNIRS to classify between three levels of the n-back task (where participants were instructed to continuously remember the last n letters of a series of rapidly flashing letters) and the rest state, reporting an average offline accuracy of 44.5%  ±  10.0% across ten participants. Weyand et al [46] investigated different combinations of six cognitive tasks and reported an offline ternary accuracy of 60.5%  ±  6.0% across ten participants. Recently, Schudlo et al [44] reported one of the first online 3-class fNIRS-BCIs. The three tasks included verbal fluency, Stroop task and rest and were differentiated online with an accuracy of 74.2%  ±  14.8% across 11 participants.

Using a BCI for online classification of brainwaves when participants mentally think yes or no has been limited to two studies, one with EEG [17] and one with fNIRS [28]. Both studies reported ~70% average binary classification accuracies (see section 1 for a more extensive summary of these BCIs).

In terms of the average classification accuracy across participants, our results surpassed the outcome of all previous mentioned BCIs except [44]. However, the Stroop task used in [44] required users to attend to a screen which is not practical for individuals with visual impairments.

Our results exhibited higher standard deviation across participants, namely, 18.3%, 17.8% and 24.7% in the last three online blocks, compared to previously mentioned BCIs. The standard deviation increased in the last block since most participants obtained higher accuracies (e.g. 100% for the last online block of P6) as the session progressed while a few hovered at chance level accuracies across all blocks (e.g. 25% in the last online block for P5 and P11). If we were to exclude the participants who were unable to complete the session without interruptions, the accuracy in the last block would jump to 81.5%  ±  13.5%. This new standard deviation is in the same range as those previously reported for multiclass BCIs. Note that P5 and P11 had difficulties obtaining accuracies above chance in both sessions, and P7 experienced difficulties at the beginning of session two. This is not unusual as certain individuals may have difficulties performing certain tasks or using certain BCI modalities (sometimes referred to 'BCI illiteracy' [47]).

In figure 4(a), we see that the channels in the left temporal and left temporoparietal regions yielded the highest Fisher criterion value, and therefore provided the most discriminative information. Channels five, seven and two provided the three highest average Fisher criterion values (13.02, 10.80 and 10.33, respectively). Channel five is located between CP5 and TP7, while channel seven is positioned between CP5 and C5, and channel two is situated close to CP5. Although the exact Brodmann areas of these channels cannot be determined without an fMRI scan, previous concurrent EEG-fMRI studies can provide an estimation of the associated Brodmann regions of these channels. Based on the channel maps from [48], these three channels (five, seven and two) approximately cover parts of Brodmann areas 21, 22, 39, 40 and 42. These Brodmann areas represent in part, Wernicke's area, the left angular gyrus, the left supramarginal gyrus and the left auditory cortex. All these areas are belong to the speech network of the brain [49] and have been previously identified in other imagined speech studies as yielding discriminative information [19].

Another channel of note is channel 20, as it is close to F7 and Broca's area (see figure 1) [48]. Although Broca's area is known to play an important role in speech production, the average Fisher criterion of this channel was 5.80, ranking it 20th out of 44 channels. This finding is in line with several previous studies on the classification of imagined speech, where greater discriminative information was found in the temporal and temporoparietal regions close to Wernicke's area, compared to Broca's area, especially when the imagined speech task did not involve the production of complicated phrases [17, 19].

Figure 4(b) depicts the brain map of the standard deviation of the calculated Fisher criterion across all participants. Again, channel five provided the highest standard deviation of the Fisher criterion. The value of the Fisher criterion varied from 0.31 (P12) to 64.33 (P6) for this channel. The large variations in the Fisher criterion values of the speech-related regions may be attributable to inter-individual performance variations of the imagined speech task. As mentioned, all participants were instructed to think yes or no while covertly repeating the phrase without any motor movements. Since they were given the freedom to 'think' yes or no in their own way, some individuals may have focused on the meaning of the phrases (affirmative versus negative response), the articulation of the phrases (with or without motor imagery of the articulation), or on imagining hearing the phrases, while covertly repeating them. For example, the highest Fisher criterion for participant 12 was obtained on channel eight (located approximately between C3 and C1) with the value of 7.09, while channel five produced the lowest Fisher criterion at 0.31. The area covered by channel eight is known to be activated during motor imagery tasks, which may indicate that this participant mainly focused on imagining motor movements required for speech production. The low accuracy of P12 compared to other participants seems to support the hypothesis that P12 may have utilized a unique approach to covert speech.

The variation across participants in the location of the channels which provided the maximum discriminative information for classification has been frequently reported in previous imagined speech studies [8, 19], and more generally, in most active BCI tasks [2]. Other than participant-specific performance of active mental tasks, this inconsistency could also, in part, be attributed to inter-individual variation in the shape and size of various brain regions. fMRI or similar imaging techniques could be used to confirm the brain regions interrogated at each of the 10–20 locations. Therefore, without the use of fMRI and structural data for each individual, it is not possible to assign a 10–20 location to a specific brain region and make a claim about the performance of a specific brain region [48].

In order to illustrate how [HbO] changed during a trial in channel five, which provided the highest average and standard deviation of Fisher criterion and was approximately the closest channel to Wernicke's area [48], a graph illustrating [HbO] versus time, averaged over all trials of each participant is shown in figure 5. Individualized activation patterns were elicited in 'yes', 'no' and rest trials, which may be attributable to participant-specific performance of imagined speech. Unsurprisingly, the difference among the three trial types was generally more visually discernable in the data of participants with the highest classification accuracies.

Zoom In Zoom Out Reset image size

As seen in table 1, the last online block yielded the highest average accuracy across participants, which was significantly higher than the first online block (p  =  0.022; as determined by a Friedman test on the accuracies from different blocks and post-hoc Wilcoxon signed rank tests on the accuracies of every pair of blocks with a Holm–Bonferroni correction for multiple comparisons). This increase in average accuracy is likely the combined effect of two factors: improved classifier robustness due to the accumulation of training data and more consistent task performance (and hence brain signals) by the user upon receiving real-time feedback [1]. The important role of feedback in improving the performance in BCI training has been reported in literature [50, 51]. Moreover, Schultz et al emphasized the importance of real-time feedback in BCIs based on speech [8].

At the beginning of the second session, there was a drop in the average accuracy. As the classifier was trained using data from a different day, this decline in accuracy may be attributable to slight variations in fNIRS cap positioning, changes in mental states between sessions (e.g. fatigue or attention [52]), or variations in metabolic states [53].

Regularization can be necessary in classification models to preserve generalizability, especially when the number of samples are of the same order of magnitude as the number of features [54]. To determine whether using a regularization parameter was helpful for online classification, we retrospectively calculated the accuracies for all online blocks without any regularization. As evident in figure 6, regularization improved the average accuracies across all blocks across participants (p  <  0.01), as suggested by a two-way repeated measures ANOVA with accuracy as the dependent variable and regularization (i.e. regularized versus not regularized) and block number as independent factors. However, when the classification accuracies with and without regularization were compared separately for each block, only the first three online blocks exhibited a significant difference (p  =  0.003, 0.032 and 0.044 for the first three blocks, respectively, using Wilcoxon signed rank test and Holm–Bonferroni correction for multiple comparisons). The significant difference in early blocks confirmed the importance of regularization when the training dataset is relatively small, as well as when a BCI is trained only on the data from a previous day. The difference was not significant in the remaining blocks, the last three online blocks, due to the inclusion of more same-day data in the classifier training.

Zoom In Zoom Out Reset image size

The selected gamma values for all participants in different blocks as well as the changes across blocks can be found in figure 7. Note that, as stated in the section 2.4.5, during each session, we restricted the upper limit of the search range for the optimized gamma to the selected gamma in the previous block. So, across each session, the selected value of gamma could either remain unchanged or decrease. As seen in figure 7, for all participants except one, the chosen gamma value decreased throughout each session most probably due to the increase in the number of training examples. Having more training examples decreases the risk of overfitting, and hence smaller regularization parameters can yield higher cross-validation accuracies.

Zoom In Zoom Out Reset image size

The control task in this study was an 'unconstrained rest'—participants were only asked to refrain from performing the other two imagined speech tasks during these trials. The admission of an idle state facilitates potential asynchronous implementation. As such, for users who achieved reliable 3-class accuracy (such as P2 and P6), imagined speech might eventually be exploited in a 2-class asynchronous BCI, where the BCI can be activated by mentally repeating the phrases 'yes' or 'no'. Such an asynchronous BCI could be used as a binary switch for an assistive device (i.e. with two activation modes) where the user might could call his/her caregiver, or start a music player, for example, without additional prompts. Using 'yes' or 'no' mental tasks to activate an assistive device may not be intuitive, but these tasks can be easier to perform than those commonly invoked in fNIRS-BCIs (e.g. mental arithmetic) [17].

Depending on the application and preference of each user, the sensitivity and specificity levels for each activation task can be tuned. For example, if the task is of high importance, such as activating a call bell for assistance, the user may prefer a high sensitivity setting to err on the side of caution. On the other hand, if accidental activations are unwanted, such as switching on and off a music player, a higher specificity may be warranted.

It should be noted that even for participants with highly reliable performance with the synchronous BCI, further training sessions within an asynchronous paradigm are required to gauge the feasibility of a truly user-paced imagined speech BCI. The development of such a BCI might proceed with only one imagined speech task to activate the BCI (like an on/off switch) and if deemed reliable after a few sessions, a second imagined speech task could be added.

For future studies, the authors suggest the use of additional sessions, as the increased number of trials may enhance classifier performance. This study demonstrated that during the second session, the average accuracies during the last online block were significantly higher than those of the first online block, which is possibly due to the increased training data. Additional sessions would shed further insight on the achievable classifier performance and robustness. Also, prior to clinical translation, the findings herein must be replicated with individuals who present as locked-in.

Future research could also explore additional BCI-specific of additional BCI-specific intuitive commands, such as 'left', 'right', 'stop' and 'go' for navigation. Using words other than 'yes' and 'no' will also reveal if the classification results obtained herein were mainly due to users' intention to provide an affirmative versus a negative response, or due to the difference between covert articulation of 'yes' and 'no'. Also, for future imagined speech BCI studies on able-bodied participants, an ultrasound system could be used to detect and discard trials which may contain possible motor confounds associated with subvocalization.

Finally, as each modality has been individually applied to the classification of imagined speech, a combination of EEG and fNIRS may exploit the advantages of each modality, potentially leading to improved BCI performance.