TempoFormer: A Transformer for Temporally-aware Representations in Change Detection

A fundamental concept underpinning longitudinal tasks is that of timelines, $P$, defined as chronologically ordered units of information between two dates (Tsakalidis et al., 2022b), here either in the form of a sequence of users’ posts, a conversation or an online thread. Specifically the $c$-th timeline, $P^{c}$, consists of a series of posts²²2We use terms posts and utterances interchangeably as the exact nature of the textual unit depends on the specific task., $u_{i}$, each with a corresponding timestamp, $t_{i}$. $P^{c}=[\{u_{0},t_{0}\},\{u_{1},t_{1}\},...,\{u_{N-1},t_{N-1}\}]$. The length of the timeline, $N$, can vary. We formulate the problem of assessing textual units in a timeline as early-stage, real-time classification, following Tseriotou et al. (2023). We map each timeline into $N$ training samples, that we call streams. Each stream contains a predefined window, $w$, of the most recent posts and a label for the most recent post: $([\{u_{i-w+1},t_{i-w+1}\},...\{u_{i-1},t_{i-1}\},\{u_{i},t_{i}\}],l_{i})$.

Fig. 2 provides an overview of TempoFormer. Its hierarchical architecture consists of three main modules, temporally-aware enhancement in multi-head attention to model the temporal distance between posts and a classification head. The modules are: post-level (local) encoding (§3.3) – obtaining word-level representation of each post using BERT’s first 10 layers; stream (global) encoding (§3.4) – modeling the sequential and temporal interactions between posts; and context-enhanced encoding (§3.6) – fusing stream-awareness in post-level representations to make them context-aware.

Each training instance is a stream consisting of the current post and its recent history, alongside corresponding timestamps: $[\{\mathbf{u}_{i-w+1},t_{i-w+1}\},...\{\mathbf{u}_{i-1},t_{i-1}\},\{\mathbf{u}_{i},t_{i}\}]$, with a total of $w$ posts in a stream. Timestamps are ignored at this stage. This stream of posts is converted into a stream, $e$, of word-level embeddings of word sequence length $K$ via the word and position embedding layer of BERT: $[\{\mathbf{e}_{1,i-w+1},\mathbf{e}_{2,i-w+1}...,\mathbf{e}_{K,i-w+1}\},...$ $\{\mathbf{e}_{1,i},\mathbf{e}_{2,i}...,\mathbf{e}_{K,i}\}]$. Specifically, in this module, the posts of each stream pass without post-post interactions via the first 10 BERT layers, resulting in hidden word-level representations for each post. Note that since a post is part of multiple streams through their window, it will pass through the BERT layers as part of each corresponding stream. For each post $j$ (belonging to a stream $q$), the word-level representations from the $z$-th Transformer layer are denoted as: $\mathbf{H}^{z}_{j_{q}}$ Therefore at the 10-th layer we reconstruct the stream and form local stream representation: $[\mathbf{H}^{10}_{i-w+1},...,\mathbf{H}^{10}_{i}]$.

Inspired by Wu et al. (2021), who model long documents hierarchically by stacking transformer-based layers of sentence, document and document-aware embeddings, we build stream and context-enhanced layers on top of post-level representations. At the stream encoding layer, we capture inter-stream dynamics. Stream-level position embeddings (PE), $\mathbf{s}^{10}$, added after the 10-th layer, encode post order within the stream. By then passing the word-level stream PE representations to another BERT layer, we obtain word-level sequence-aware updated hidden representations $[\mathbf{H}^{11}_{1,i-w+1}$, …,$\mathbf{H}^{11}_{1,i}]$.

Next, we obtain the order-aware [CLS] token from the stream and apply Temporal Rotary Multi-head Attention (MHA), a proposed variation of RoFormer (Su et al., 2024), which accounts for the temporal rather than the sequential distance between posts (see §3.5). These context-aware, temporally-enhanced tokens are fed back to replace the respective [CLS] tokens in the hidden representations from the previous BERT layer, resulting in $[\mathbf{H}^{{}^{\prime}11}_{1,i-w+1}$, …,$\mathbf{H}^{{}^{\prime}11}_{1,i}]$. This enables the propagation of the learnt stream embeddings to the post-level.

Here, in order to model the temporal dynamics, we propose a novel variation of Eq. 1, named Temporal Rotary Multi-head Attention, making use of the relative position property. Instead of $R^{d}_{\theta,n-m}$, we reformulate the rotary matrix to model the temporal, rather than the positional differences, $R^{d}_{\theta,\mathbf{t_{n}-t_{m}}}$. We employ it on the stream-level using the [CLS] tokens to capture the stream global context through both the temporal and linguistic dynamics. The developed layer includes solely self-attention without the need for feed-forward and normalisation layers. In practice, since we measure time in seconds, we log-transform time in order to remove task dependancies on the scale of temporal propagation, to account for stream non-linearities and to alleviate exclusion of temporal outliers.

Literature has shown the effectiveness of enhancing word-level representations hierarchically through context-level learnt dynamics (Zheng et al., 2020; Wu et al., 2021; Ng et al., 2023). To this effect we introduce a second-layer of stream-level position embeddings, $\mathbf{s}^{11}$, to re-instate the absolute sequence position of each post for context-enhanced modeling. These are fed into a global context-aware layer, essentially a word-level transformer layer. Since the [CLS] tokens of each post are stream-aware, they contextualise the token-level representations based on the temporal and global learnt dynamics, obtaining: $[\mathbf{H}^{12}_{1,i-w+1},...,\mathbf{H}^{12}_{1,i}]$. To fully model the stream dynamics given the now context-enhanced [CLS] tokens, we employ a last layer of Temporal Rotary MHA resulting in $[\mathbf{H}^{{}^{\prime}12}_{1,i-w+1},...,\mathbf{H}^{{}^{\prime}12}_{1,i}]$. Lastly, we adapt the Gated Context Fusion (Gate&Norm) mechanism by Zheng et al. (2020) to fuse both the utterance word-level informed ($\mathbf{H}^{12}_{CLS}$) and the stream utterance-level informed ($\mathbf{H}^{{}^{\prime}12}_{CLS}$) [CLS] tokens through element-wise multiplication $\odot$:

Although the proposed architecture can in principle be applied to any Transformer-based model, we select BERT (Devlin et al., 2018) as the foundation model and initialise all word-level weights. Literature on longitudinal context-aware classification has shown the importance of efficiently combining the current utterance with historical information (Sawhney et al., 2020, 2021a; Tseriotou et al., 2023). We thus concatenate the local stream-agnostic [CLS] token of the current utterance from the 10-th layer, $\mathbf{C}^{L}_{CLS}$, (obtained through typical BERT Pooling) with the obtained global stream-enhanced [CLS], $\mathbf{C}^{G}_{CLS}$ (Fig. 2). This final representation is fed through two fully connected layers with ReLU activation and dropout (Srivastava et al., 2014). The architecture is fine-tuned for each classification task (§4) using alpha-weighted focal loss (Lin et al., 2017), to assign more importance to minority classes and alleviate class imbalance.

We test our model on three different longitudinal tasks of different temporal granularity: 1) Stance Switch Detection – identification of switches in overall user stance around a social media claim, 2) Moments of Change (MoC) – identification of mood changes through users’ online posts and 3) Conversation Topic Shift – conversation diversion identification. We adopt a real-time prediction formulation (see §3.1) to assess system ability to perform early change detection in real-world scenarios. Table 1 provides the detailed statistics of each dataset showing the different degree of temporal granularity and dataset specifics. More details are provided in Appendix A.

We select baselines that are both post-level and stream-level (current post and recent history window, see §3.1).

We present results for TempoFormer and baselines in Table 2. TempoFormer is the most performant in all three tasks based on macro-averaged F1. We note that recurrent models based on pre-trained BERT representations (BiLSTM for LRS and Topic Shift MI and Seq-Sig-Net for TalkLife), ranked second best. These models have been the SOTA for these datasets (Tseriotou et al., 2023, 2024). While the datasets are of different sizes, temporal characteristics, timeline length and change event distribution (see Table 1), TempoFormer retains its high performance, showcasing its generalisability for real-time change detection. Importantly, our model has the highest F1 for all minority classes, with the exception of Topic Shift MI, where other baselines have higher class-specific F1 scores for M but much lower F1 for R. Since TempoFormer operates on a contextual window of recent posts we select the appropriate window for each stream based on a window analysis, reported in §5.2.

We distinguish baselines into post and stream-level ones, noticing that smaller fine-tuned Language Models, even as simple as an FFN, allowing for stream-level context, score consistently better than post-level ones - with the exception of RoBERTa for TalkLife. This consistent finding underscores the importance of developing contextually informed representations for change detection. Few-shot prompted LLMs have consistently lower performance than smaller fine-tuned LMs, in line with reported poor temporal abilities of LLMs (Jain et al., 2023; Bian et al., 2023). For post-level, while Mistral’s performance improved from 5 to 10-shot, it is still barely above the random baseline and significantly behind BERT and RoBERTa. For LRS and Topic Shift MI the stream-level 5 and 10-shot Mistral performance increases, but falls way short of BERT/RoBERTa and all the stream-level models, indicating that although sequential context is important it is not modelled appropriately using LLMs. In line with (Wenzel and Jatowt, 2024), Llama2 suffers from generating responses outside the predefined classes, resulting in very low performance. TempoFormer demonstrates a generalisable architecture that enhances word-level post representations given the context, while modelling effectively the interplay between linguistic and temporal dynamics.

In Table 3 we present an ablation study to assess each of TempoFormer’s components.

Since longitudinal and change detection models have so far heavily relied on recurrence-based architectures, we evaluate the effect of recurrence on models jointly trained for stream and post-level representations. To do so we adapt RoBERT (Pappagari et al., 2019), originally developed for long document classification, which applies recurrence over BERT’s pooled [CLS] for each post. Here, we propose RoTempoFormer, a modification of RoBERT, that uses recurrence over TempoFormer’s pooled [CLS] for each post. Both RoBERT and RoTempoFormer are stream-level, recurrence-based models. We report results in Table 4.

RoTempoFormer consistently outperforms RoBERT for all datasets. RoTempoFormer strikes the right balance between jointly modeling context-aware post representations and recurrence in stream dynamics. Only for LRS do recurrence-based models have a better performance than TempoFormer. To examine this phenomenon we measure the diversity of each dataset with respect to its content and report it in Table 5. We report the BERTScore (Zhang et al., 2019b) and Cosine similarity between SBERT pairs of representations as well as the Outlier metric (Larson et al., 2019; Stasaski et al., 2020) on SBERT which measures the Euclidean distance between the (unseen) posts in the test set and the mean training corpus across folds and seeds for all datasets. Thus we assess both the semantic diversity and test set diversity. Across all metrics we consistently see that TalkLife is the most and LRS the least diverse. We postulate that for more diverse datasets like TalkLife, RoBERT has a really low performance, while it performs much better on less diverse ones. This could be due to: 1) overfitting due to recurrence and 2) inability of RoBERT to jointly model diverse context-aware representations, while capturing their evolution. RoTempoFormer, maintains its high performance, striking a good balance between modeling the context-aware post-level and the timeline-level dynamics. Importantly, we show that TempoFormer can be used as the foundation for temporal representation learning in other architectures.

To examine the flexibility of the TempoFormer stream-level and context-enhanced layers beyond the BERT architecture, we use TempoFormer with RoBERTa (roberta-base). Specifically, we allow the first 10 RoBERTa layers to model post-level (local) dynamics and modify its top two layers to capture stream dynamics. Since in Table 2, TalkLife benefits from the use of RoBERTa over BERT at the post-level, we examine if this gain also transfers to the TempoFormer. Summarising results in Table 6, we show that the RoBERTa-based TempoFormer achieves a new SOTA of 58.8% macro-avg F1, +1.7% over the BERT-based TempoFormer. This increase is in line with the +1.5% performance increase between vanilla BERT and RoBERTa macro-avg F1. Importantly, the increase in overall F1 is driven by clear performance gains in the IE and IS minority classes, further demonstrating the success and adaptable nature of TempoFormer in identifying changes over time.

While TempoFormer shows SOTA performance on three different tasks and datasets of diverse temporal granularity involving change detection, namely: social media overall stance shift, user mood change detection and open conversation major topic shift detection, we are yet to evaluate its performance on a wider range of tasks and datasets. Additionally, although we demonstrate strong performance in datasets as small as  5,500 data points, we believe that our model, as most machine learning models, benefits from larger corpora in training where we can more meaningfully fine-tune the inter and intra-post relationships to model the dataset’s linguistic style and change intricacies. TempoFormer models post dynamics through a predefined stream window, identified through understanding the characteristics of a dataset via preliminary experiments. The need for initial exploration can be limiting compared to a dynamic window setting. Furthermore, despite the fact that our implementation is flexible and can be applied to different encoder architectures, the codebase is built in PyTorch, therefore imposing the constraint of PyTorch-only frameworks. On the classification front, we operate on a supervised setting therefore assuming the availability of annotated data which can be expensive to obtain especially from experts. Regarding evaluation, we focus on post-level metrics, and have not yet considered metrics more appropriate for longitudinal tasks and streams Tsakalidis et al. (2022b). Lastly, since our model operates by fine-tuning a pre-trained transformer-based model, like BERT, it automatically assumes the availability of such model in the language of the dataset/interest (English in our case), which might not be the case for low-resource languages.

The performance of our model, TempoFormer, is demonstrated on three datasets: LRS, TalkLife and Topic Shift MI. The LRS dataset is based on the publicly available RumourEval 2017 dataset (Gorrell et al., 2019) for stance detection, while the Topic Shift MI dataset is also a publicly available dataset based on human to human open domain conversations. Since the TalkLife dataset contains sensitive and personal user data, the appropriate Ethics approval was received from the Institutional Review Board (IRB), followed by data anonymisation and appropriate sensitive data sharing procedures. Access to this dataset was granted and approved by TalkLife ³³3https://www.talklife.com/ through licensing for research purposes associated with the corresponding submitted proposal. All examples in the paper are paraphrased. Models were built on a secure server with authorised user-only access, while the labeled TalkLife dataset and the developed models are not indented to be made publicly available in order avoid any potential risks of unintended use.