MASSFormer: Mobility-Aware Spectrum Sensing using Transformer-Driven Tiered Structure

Mobility scenarios involve the movement of both PUs and SUs within the network environment. PUs mobility affects the prediction of PU states due to changes in PU activity patterns with their movements. The PU signals reaching SUs encounter varying levels of path loss and different channel conditions due to SUs’ mobility. In this work, we consider the Random Waypoint (RW) Mobility Model [22] to find the movement patterns, trajectories, speed, direction, and location information of PUs and SUs. According to the RW mobility model, PU and SUs are initially distributed randomly in the simulation area. The users wait for pause time by staying in one location, and once this time expires, they choose a random destination in the simulation area and a speed that is uniformly distributed $[v_{min},v_{max}]$. The users move to the new location at the selected speed and wait for a pause time after arrival. Fig. 1 shows the movement pattern of PU and SUs within a defined simulation area. Initially the position of moving user i.e. PU or SUs is represented as $(X,Y)$, after a time-interval $\Delta t$, the position is updated using the following formulas:

where speed $v$ is randomly selected between $[v_{min},v_{max}]$, $\theta(t)$ denotes the direction at time $t$. The position traces of PU and SUs are calculated using equation (2) and (3).

The instantaneous signal-to-noise ratio experienced by $s$-th SU is denoted as $\gamma_{s}=\frac{{|h_{s}|}^{2}Pt}{N_{0}B_{W}}=\frac{Pr_{s}}{N_{0}B_{W}}$. We assume that the transmit power of PU is fixed to $P_{t}$ and PU signals are transmitted through the channel whose bandwidth is $B_{W}$. The received PU power at SU $s$ at a distance $d_{s}(u)$ from the PU at time $u$ can be expressed in dB as

where $\alpha$ and $\beta$ represent path-loss exponent and path-loss constant respectively, and $N_{0}$ denotes noise power spectral density.

The proposed model requires labeled training dataset collected in $U$ sensing durations. The raw sensing signal $\boldsymbol{Y}_{u}^{s}$ collected during a single sensing duration is too large with dimension $M\times N$, to be directly considered as input for the neural network. Most of the existing SS methods utilized CMs as test statistics. Therefore, it is required to compress the sensing signal into CMs to construct the training set. The signal matrix $\boldsymbol{Y}_{u}^{s}$ composed of sensing signals from $M$ antennas of SU $s$ to fusion centre can be expressed as

Let us construct the CM of the signal matrix as defined below

Now, construct the training set by arranging the CMs as defined as $\{(\boldsymbol{R}^{s}_{1},b_{1}),(\boldsymbol{R}_{2}^{s},b_{2}),(\boldsymbol{R}_{3}^{s},b_{3})...........(\boldsymbol{R}_{U}^{s},b_{U})\},$ where $U$ denotes total examples in the set. The class of $u$-th example is denoted as $b_{u}\in\{0,1\}$, where the hypotheses $\mathcal{H}_{1}$ and $\mathcal{H}_{0}$ are selected when $b_{u}=1$ and $b_{u}=0$, respectively. To predict the PU states at SU-level, we concurrently analyze the CMs of sensed signals gathered at $s-$th SU during numerous sensing intervals. The dataset for $s$-th SU is written as

where, $\boldsymbol{\phi}^{s}_{u}=[\boldsymbol{R}^{s}_{\lambda(u-1)+1},\boldsymbol{R}^{s}_{\lambda(u-1)+2},....,\boldsymbol{R}^{s}_{\lambda(u-1)+\lambda}]$ and $\lambda$ denotes the length of input’s temporal sequence. The $u$-th example in the set of $s$-th SU is denoted as $\boldsymbol{\phi}^{s}_{u}$. The complete dataset for $s$-th SU is represented as ${\Phi}^{s}$. To generate data for all SUs, we follow the same method as described in equation (5), (6), and (7). Since every participating SU sends its sensing data to the fusion centre, the complete dataset constructed by collecting samples from all SUs is defined as

In this section, we discuss the tokenization process of ViViT, as we adopt a similar approach for tokenization in our methodology. A video sample is denoted as $X\in{\mathbb{R}^{T\times H\times W\times C}}$, where $T$, $H$, $W$, and $C$ referring to the temporal length, height, width and depth of the input respectively. The ViT-based model processes the $2$-D images to extract $N_{t}$ non-overlapping patches. In the context of videos, a sequence of spatio-temporal "tubes" $x_{1},x_{2}......x_{N_{t}}$, ${x}\in{\mathbb{R}}^{{t}\times{h}\times{w}}$ are extracted from the input volume as described in ViViT [20]. A linear operator $E$ is applied to the tubes to linearly project them to $1$-D tokens ${z}\in\mathbb{R}^{N_{t}\times d}$, where $N_{t}=n_{t}n_{h}n_{w}$, $n_{t}=\lfloor\frac{T}{t}\rfloor,n_{h}=\lfloor\frac{H}{h}\rfloor$, and $n_{w}=\lfloor\frac{W}{w}\rfloor$. To perform linear projection, $3$-D convolution is used with the kernel size $(t,h,w)$ and the strides $(t,h,w)$ in time, height, and width dimensions, respectively. In ViViT, a learnable token $z_{cls}$ is prepended to the token sequence. However, to incorporate the information from all the tokens, we apply attention based sequence pooling operation to compute a 1-D feature vector from 2-D encoder output. As the self-attention mechanism in the transformer encoder is order agnostic, positional embedding $P\in\mathbb{R}^{N_{t}\times d}$ is added to tokens to preserve the positional information. Therefore, we extract tokens from the sequence of CMs constructed for each of the SUs as depicted in Fig. 2. The sequence of tokens for $s$-th SU is denoted as

The SU-transformer network and collaborative transformer network are made up of several transformer encoder layers. The encoder layer consists of multi-head self-attention (MSA) and multi-layer perceptron (MLP) in sequence with skip connections. The sequence of tokens $z$ corresponding to each SU is processed by the SU-transformer network which consists of $L_{1}$ encoder layers. Each encoder layer is applied sequentially which consists of the following operations,

where LN is layer normalization [23], MSA is multi-head self attention [18] and MLP [24] consists of two dense layer separated by GeLU non-linearity. The self-attention aims to capture interactions among all $N_{t}$ token embeddings. The self-attention is computed as

The output of the self-attention block is obtained by multiplying $Z_{att}$ with the value matrix $V$. MSA allows the model to concurrently attend to information from various representation subspaces at different positions. The output of MSA block is computed as:

where

where $W^{Q_{i}}$ $\in$ $R^{d\times d_{q}}$, $W^{K_{i}}$ $\in$ $R^{d\times d_{k}}$ and $W^{V_{i}}$ $\in$ $R^{d\times d_{v}}$, and $W^{O}$ $\in$ $R^{h_{d}d_{v}\times d}$, $d_{v}=d_{k}=d/h_{d}$. $i$ denotes number of heads ranging from $1$ through $h_{d}$ and $h_{d}$ being the total number of heads in MSA block.

In this section, we provide a brief introduction to the proposed MASSFormer method. The main objective of the proposed architecture is to learn the collective PU states by selectively extracting the most relevant features from the sensing data of all cooperating SUs. The proposed MASSFormer model is divided into two components i.e. the SU-transformer network and collaborative transformer network to predict PU states by extracting spatio-temporal features at individual SU-level and at group-level respectively. The proposed MASSFormer architecture is depicted in Fig. 2, where SU-transformer network is used to predict the SU-level PU states by extracting spatio-temporal features from the sequence of tokens extracted from a series of CMs of a $s$-th SU. Further, collaborative transformer network is used to model the output representations of all SUs to predict the PU states at the group-level. Attention from various levels of a transformer can extract contributing features from the participating SUs in a progressive manner.

To predict the PU states, tokens computed from samples of each SU are fed to SU-transformer network in different parallel pipelines. SU-transformer network consists of $L_{1}$ encoder layers, followed by sequence pooling as described in section III-D and MLP block to predict the PU states at SU-level. We concatenate the output feature representations $K_{s}$ of each $s$-th SU.

Further, max pooling operation is performed over the feature $K_{s}$ of all SUs to get output representation $G_{t}$ provided in equation (15). Instead of spatial pooling (like typical max pooling in CNNs, which pools over spatial dimensions), this operation pools over the dimension representing different SU and combines their predictions by selecting the maximum value for each position across SU dimension. The output features $G_{t}$ is fed to collaborative transformer network which consists of $L_{2}$ transformer encoder layer, followed by sequence pooling and MLP block, to extract the group-level features from all the participating SU. Finally, the output features are fed to Softmax layer for predicting the activity states of PU.

For the prediction of class label, ViViT [20] forwards a learnable class token through the encoder layers and later to the MLP for classification. In contrast, we use sequence pooling, first proposed by Hassani et. al. [25] to extract representation. Sequence pooling, an attention-based method, assigns weights to sequential embeddings, transforming the input sequence into a vector representation. This involves assigning importance weights to the processed data from transformer encoder. The motivation is rooted in the dispersion of information across all tokens in the output sequence, necessitating aggregation by assigning suitable weights to each token. Sequence pooling involves the transformation of sequence as $T:$ $\mathbb{R}^{N_{t}\times d}\mapsto\mathbb{R}^{d}$. Given the output $z\in\mathbb{R}^{N_{t}\times d}$ of encoder is passed to linear layer $g(z)\in\mathbb{R}^{N_{t}\times 1}$, and softmax activation to the token sequence as follows

Hence, we calculate importance weights for each input token as described in equation (16), and further, these weights are assigned to each token as described in equation (17). The computed weights are employed to adjust the contributions of the tokens through a weighting operation as follows

$z_{seq}$ is passed to MLP block of the SU-transformer network and collaborative transformer network to detect activity states of PU at SU-level and group-level respectively.

The developed model is trained in two distinct stages. In the initial stage, the SU-transformer network is trained end-to-end using a training dataset comprising sequences of CMs converted into tokens and associated labels. This process aims to extract spatio-temporal features at SU-level. After the SU-transformer network gets trained, the output features, $K_{s}$, extracted for each SU during the initial stage are concatenated and further, a max pooling operation is applied to obtain pooled feature representation denoted as $P_{t}$. In the subsequent stage, the pooled features from multiple sensing periods are inputted into the collaborative transformer network which comprises various encoder layers, followed by sequence pooling and an MLP block.

The resulting output features are fed to the Softmax layer for the prediction of probabilities associated with hypotheses $\mathcal{H}_{0}$ and $\mathcal{H}_{1}$. The $u$-th example in the set contains data samples from each SU is denoted by symbol ${\Phi}_{u}$, where ${\Phi}_{u}=(\phi_{u}^{1},\phi_{u}^{2}...\phi_{u}^{S})$. Normalized output of the developed model is represented as $[j_{{\theta}/\mathcal{H}_{0}}(\boldsymbol{\Phi}_{u}),j_{{\theta}/\mathcal{H}_{1}}(\boldsymbol{\Phi}_{u})]$, where $j_{{\theta}/\mathcal{H}_{0}}(\boldsymbol{\Phi}_{u})+j_{{\theta}/\mathcal{H}_{1}}(\boldsymbol{\Phi}_{u})=1$. For a new data point $\boldsymbol{\Phi}_{u}$, the trained model predicts a probability associated to either Hypothesis $\mathcal{H}_{0}$ and $\mathcal{H}_{1}$. $j_{{\theta}/\mathcal{H}_{i}}(\boldsymbol{\Phi}_{u})$ denotes the class probability of hypothesis $\mathcal{H}_{i}$. Based on this, the main goal of training is to maximize the likelihood as

This is equivalent to minimizing the cost function, a cross-entropy loss function, which is defined as logarithm of the likelihood function and normalizing it with the number of training points.

To achieve the maximum likelihood, the cross-entropy loss function is minimized to obtain the optimal parameters as

While training the developed model, Back propagation (BP) algorithm is used to compute gradients of the loss function, and applying Adam optimizer to optimize the model parameters to get well-trained network. The output of trained model is denoted as $j_{\theta^{*}|{\mathcal{H}_{0}}}(\boldsymbol{\Phi}_{u})$ and $j_{\theta^{*}|{\mathcal{H}_{1}}}(\boldsymbol{\Phi}_{u})$. According to Neyman-Pearson (N-P) theorem, the optimal test statistics is likelihood ratio is represented as

Upon the arrival of a new data point $\boldsymbol{\Phi}_{u}$, the model output is used to determine the activity states of PU. However, this approach does not provide control over the false alarm probability $p_{fa}$. Therefore, we calculate the detection threshold $\gamma$ using Monte-Carlo method that ensures the desired false alarm probability $p_{fa}$ is achieved. We collect the sequence of CMs corresponding to hypothesis $\mathcal{H}_{0}$ to build a set $\boldsymbol{\Phi}_{\mathcal{H}_{0}}=\{\tilde{\boldsymbol{\Phi}}_{1},....,\tilde{\boldsymbol{\Phi}}_{\tilde{Q}}\}$, where total examples in set $\boldsymbol{\Phi}_{\mathcal{H}_{0}}$ denoted by $\tilde{Q}$. Now, we fed the noisy data samples to the MASSFormer and get the expression of test statistics under $\mathcal{H}_{0}$ i.e. $\beta_{MASSFormer|{\mathcal{H}_{0}}}$. Further, reorganize the all these $\beta_{MASSFormer|{\mathcal{H}_{0}}}$ to construct a set $\tilde{\boldsymbol{\Phi}}_{MASSFormer|\mathcal{H}_{0}}$ in ascending order such as $\forall 1\leq p\leq q\leq\tilde{Q},$

Lastly, we compute the detection threshold with constraint of $p_{fa}$ as

The function $round(.)$ provides the nearest integer value of a given number. For a test data, a decision about the activity states of PU can be made as follows

We consider the simulation area of $1000$m $\times$ $1000$m area, and they move with a velocity randomly chosen between $[20,25]$ m/sec, resulting in dynamic changes in their positions over time. The pause time is chosen as $1$msec. This setup allows us to assess the feasibility of the proposed model under conditions of user mobility. We assume that PU signals are independent and identically distributed (i.i.d.) Gaussian Random vector with zero mean and signal variance $\sigma_{w}^{2}$. We have chosen values of $N=100$, $S=3$, and $M=15$, length of temporal sequence $\lambda=20$ randomly. PU signals are assumed to be sent on a channel having bandwidth $B_{W}$ is $10$MHz. Furthermore, we assume that $P_{t}=200mW$, $\beta=10^{3.453}$ and $\alpha=3.8$, and noise power density is $N_{0}=-150dBm/Hz$. We assumed that the noise signal represented by $\eta^{m}_{s}(n)$ and $\eta_{c}(n)$ follow an i.i.d Gaussian random vector with zero mean and noise variance $\sigma_{\eta}^{2}$. To address the challenges occurring due to user mobility in real-world scenarios, we assumed that PU signals reaching from multiple SUs to fusion centre may experience different path loss and fading severity. Consequently, we assumed that channel gains $h_{s}^{m}(n)$ and $h_{s,r}^{m}(n)$ follow Rayleigh distributions and fading severity is modeled by adjusting the fading parameter. The received signal power is calculated using equation (4), which incorporates path-loss between PU and each SU. The hyperparameters used for our proposed architecture are provided in Table I. The number of training and testing points used for training and evaluating the proposed models are $104,000$ and $15,000$. The number of epochs and batch size are considered to be $100$ and $16$.

We conducted comprehensive simulations to illustrate the efficacy of the proposed MASSFormer method. We evaluated the detection performance in terms of receiver operating characteristics (ROC) curve, probability of detection ($P_{d}$) vs noise power density $(N_{0})$, and sensing error vs noise power density. We compared and analyzed the performance of MASSFormer method with existing methods such as CNN-LSTM [16], APASS [9], and $3$D CNN [21] methods. We rigorously evaluated the performance of the proposed method under various scenarios, including imperfect sensing as well as both perfect and imperfect reporting channels, to thoroughly test its robustness. We calculated the $P_{d}$ values at different detection thresholds computed at different values of $p_{fa}$ to evaluate the detection performance. Fig. 3 shows the ROC curve of the proposed MASSFormer method along with existing methods at two different noise power density values such as $N_{0}=-150dBm/Hz$ and $N_{0}=-145dBm/Hz$.

The results show that the proposed MASSFormer method performs better as compared to methods CNN-LSTM, APASS, and $3$D CNN in terms of detection performance.

In accordance with IEEE 802.22 standard, the maximum acceptable value of false alarm probability is $(P_{f}\leq 0.1)$. Therefore, we provide a plot of $P_{d}$ versus noise power density at a fixed value of false alarm probability $P_{f}=0.09$ in Fig. 4. From Fig. 3 and Fig. 4, we observe that the detection probability decreases as the noise power density increases. The sensing error, which is defined as the mean of the probability of miss detection and probability of false alarm, another performance metric, is used to evaluate the performance of the proposed method.

Fig. 5 depicts a plot of sensing error vs noise power density for different methods. We observe that the sensing error increases with increasing noise power density, reflecting inaccuracies in individual SUs’ sensing outcomes. After analysis, we find that the proposed method achieves lower sensing error than the existing methods, highlighting its benefits. Fig. 6 depicts the plot of classification accuracy vs noise power density for different methods. After a comprehensive analysis of the results, it is evident that the proposed MASSFormer method outperforms the existing methods, exhibiting superior detection performance in terms of detection probability and classification accuracy.