VCHAR:Variance-Driven Complex Human Activity Recognition framework with Generative Representation

In line with the development of robust foundation models, our approach involves pretraining a sensor-based foundation model encapsulating a comprehensive suite of elements common to a wide array of scenarios. This model serves as a versatile starting point, designed to be fine-tuned subsequently to accommodate the specificities of distinct scenarios encountered by users.

The CHAR foundation model is imbued with a rich lexicon that describes a multitude of complex scenarios and human activities. It not only captures the essence of activity patterns but also identifies sensor-specific signatures that are pivotal for accurate activity detection. This capability ensures that, upon detection, the most relevant sensors are highlighted, providing intuitive visual cues within the generated explanations.

For instance, the foundation model is pretrained with elaborate activity narratives such as ”morning routine”, ”making a sandwich”, or ”preparing cereal”. These narratives embody complex sequences situated within a broader context of atomic activity representations. Moreover, the model is attuned to discern and emphasize sensors that yield significant information during the activity recognition process. This level of detail furnishes users lacking domain expertise with rich, contextual insights into the sensor data and the detected activities.

The foundation model thus constitutes a preparatory step in building an adaptable, knowledgeable system for CHAR. Through fine-tuning, it can swiftly assimilate into any designated smart space environment, delivering bespoke explanations tailored to the unique configuration and operation of that space.

During the pretraining phase, the foundation model’s objective function aims to minimize the expected discrepancy between the model’s predictions and the ground truth. The foundation model processes three distinct inputs: the visual content $V$, represented by images or videos; the query $Q$, which contextualizes the visual content; and the sensor encoder output $E$, which encapsulates the sensor-derived information. The objective function is thus characterized by the following minimization:

Here, $\Psi_{\theta}$ denotes the predictive function of the foundation model parameterized by $\theta$, GT represents the ground truth corresponding to the inputs, and $\mathcal{D}$ signifies the joint distribution of the visual, query, and sensor data. This optimization ensures that the foundation model is capable of integrating and interpreting multimodal inputs to generate a comprehensive and accurate depiction of the observed activities in preparation for fine-tuning to specific smart environment applications.

In our approach, $E$ symbolizes a set of features including the type of activities identified, the values of sensor activation, among others. To enhance interpretability, we convert these features into a structured prompt using a predefined template. For instance, given an input vector $[0.1,0.2,0.8,0.5]$ where the third value indicates the highest sensor activation value from the model, and knowing that the corresponding sensor is located on the arm, our template highlights this sensor in yellow within the generated video. Concurrently, the model identifies the complex activity, such as ’making coffee’, and a atomic activity, such as ’opening a door’. These elements are woven into a cohesive prompt: ”Someone is opening the door ,complex activity ”making coffee” ”. This narrative integration allows for a user-friendly representation of activities, facilitating a clear and intuitive understanding of the ongoing events as detected by the sensor system.

When transitioning our model to specific scenarios, it is common to encounter variations in the manifestation of activities and their corresponding complex contexts. To accommodate these scenario-specific nuances, we adopt a ”one-shot tuning” strategy(Wu et al., 2023). This strategy rapidly recalibrates our sensor-based foundation model to align with the new scenario characteristics. This is particularly pertinent for incorporating new activity videos where the contextual dynamics may significantly differ.

To further enhance the model’s adaptability and descriptive capabilities, we introduce a masked training strategy. This approach facilitates the model’s ability to generalize across diverse descriptive modalities. During fine-tuning, carefully designed prompts address the functionalities of specifically masked regions within the video. These prompts serve a dual purpose: they guide the video generation process in the latent space and ensure that the model’s output is both representative and specific to the newly adapted scenario.

Employing this method enables the foundation model to not only recognize and adapt to new scenarios but also to generate activity representations that are rich in detail and contextually relevant.

Latent diffusion models (LDMs)(Rombach et al., 2022a) are pivotal in our approach, functioning directly within the latent space of new video embeddings. These models utilize an encoder to project videos into a latent space, facilitating manipulations within this reduced dimensionality before reconstruction. Specifically, an encoder $\mathcal{E}$ maps a video $V$ onto a latent representation $\mathbf{z}=\mathcal{E}(V)$, and a decoder $\mathcal{D}$ subsequently restores this latent representation back to the video domain, approximating the target video, i.e., $\mathcal{D}(\mathcal{E}(V))\approx V$.

In our case, when new activity types are introduced to the model, the LDMs adeptly handle the designated masked regions within the latent embeddings. The decoder $\mathcal{D}$ is then employed to regenerate the video, ensuring that it accurately reflects the complex activity type and sensor activation value in side of the model.

DDIM, the Denoising Diffusion Implicit Model, is integral to enhancing the efficiency of the video generation process, particularly in maintaining the structural integrity of the activity movements(Wu et al., 2023; Song et al., 2020).In our implementation, the DDIM operates within the latent space of a new video’s embedding. The process utilizes prompts that encapsulate masked patterns, denoted by $T$, alongside a guiding prompt $T^{*}$ for directed generation. The final video $\mathbf{V}$ is produced by first inverting the latent embedding via $\text{DDIM}_{\text{inv}}$, using the encoder’s output informed by the mask pattern and guiding prompt, followed by sampling with $\text{DDIM}_{\text{samp}}$. This sequence can be represented mathematically as:

Here, $\mathcal{E}$ signifies the encoding function, $\mathcal{D}$ the decoding function, and $\mathbf{V_{0}}$ the initial video input. The inclusion of both $T$ and $T^{*}$ ensures that the DDIM not only captures the masked patterns but also adheres to the specified activity prompts. This dual guidance mechanism facilitates precise control over the generative process within the latent space, yielding an output video that accurately reflects the desired activities.By operating in this fashion, our model not only retains the fidelity of the original video but also enriches it with detailed descriptions of activities, providing a comprehensive depiction that encompasses both the observed and inferred aspects of the scene.

Masked training can markedly expedite the training of substantial diffusion models such as Stable Diffusion(Zheng et al., 2023), without detriment to their generative capabilities. To hasten the learning process, we employ a masked strategy tailored to specific patterns desired for our sensor descriptions. For instance, as depicted in Figure 6, the output frame rate is set at 24 fps, enabling us to direct the model’s output toward varying scenarios. Additionally, a human study is conducted to further evaluate the effectiveness of these strategies.

Our decoder leverages the stable diffusion model (Rombach et al., 2022b), utilizing online pretrained weights. Despite this, specific patterns crucial for representing complex scenarios in smart spaces are absent, necessitating further pretraining of these missing concepts post-weight loading. Our pretraining involves adjusting all model weights, whereas finetuning focuses only on several attention layers to employ a one-shot tuning strategy(Wu et al., 2023), significantly reducing GPU memory usage. This efficiency allows the use of standard gaming GPUs, like the RTX series.

Specifically, the complete model requires 10,000 steps for pretraining to incorporate new features, and only 500 steps to fine-tune for video applications, with a batch size of one. For inference, the DDIM sampler requires 100 steps. We utilize an NVIDIA A100 GPU for pretraining and an RTX 6000 for fine-tuning. The entire pretraining process takes approximately 48 hours to integrate a new feature, while finetuning is completed in just 30 minutes. This rapid adaptability is particularly beneficial for individualized smart space applications, where conventional GPUs can effectively handle the adaptation of new features to the model. Such capabilities render our framework practical for real-life implementations.

The Opportunity dataset is a publicly accessible benchmark for human activity recognition algorithms, featuring data from 4 subjects out of an original 12 (Ciliberto et al., 2021) . It includes 15 networked sensor systems with 72 sensors across ten modalities embedded in the environment, objects, and worn on the body. This study particularly focuses on inertial sensors placed on the left lower arm, left upper arm, right lower arm, right upper arm, and back of the torso, which record data via accelerometers, gyroscopes, and magnetometers.

We analyze 17 micro-activities and 4 complex activities that offer a comprehensive view of human motion (table 1). The dataset’s design ensures a realistic portrayal of activities, with annotations at multiple abstraction levels suitable for testing advancements in sensor selection, feature extraction, classifier training, multimodal data fusion, segmentation, and hierarchical activity recognition. This rich framework makes the Opportunity dataset an excellent resource for evaluating the effectiveness of different activity recognition strategies under naturalistic conditions.We use 3 subject as training set, 1 subject as testing set.In this study, we utilize a 20-second sliding window approach for data processing, which frames our training set with dimensions of (3036, 72, 600) and our testing set as (1299, 72, 600). This strategy is instrumental in capturing the temporal dynamics essential for our activity recognition tasks.

The FallAllD dataset is a specialized resource tailored for research in fall detection, fall prevention, and human activity recognition, suitable for both classical and deep learning methodologies. Data collection involved three identical data-loggers worn by participants on the neck, wrist, and waist, each outfitted with an inertial module (comprising an accelerometer, gyroscope, and magnetometer) and a barometer.

Data were collected from 15 participants aged between 21 to 53 years old, resulting in 26,420 files, each 20 seconds in duration. Within this dataset, complex scenarios such as ”fall” and ”no fall” detection are highlighted, where atomic activities are indicative of both scenarios. For example, activities might include normal actions such as walking and sitting, as well as the same actions performed with an accidental fall. For our studies, we selected 8 (table  2) activities that span both complex categories, allocating around 20% of the data for testing and 80% for training purposes.In our methodology, we adopted a 10-second sliding window for data segmentation, resulting in a training set dimensionality of (4964, 12, 4760) and a testing set dimensionality of (1375, 12, 4760). This windowing technique is critical for capturing temporal patterns in the sensor data conducive to recognizing complex activities.This dataset does not include specific labels for atomic activities. Each file only provides information on the type of activities, device type, recording date, and subject number.

The Cooking Activity Recognition Challenge dataset(Lago et al., 2020) is a multifaceted collection of sensory data, recorded via smartphones, wristwatches, and motion capture systems, designed specifically for the complex task of activity recognition. This dataset, procured from 4 subjects, comprises accelerometer data from the right arm, left hip, and both wrists, in addition to motion capture information from 29 distinct markers. It categorizes activities into three macro activities—making a sandwich, preparing a fruit salad, and cereal preparation—alongside 10 micro activities such as adding, cutting, mixing and ”other”. In our recognition task, we focused on nine specific activities that were relevant to our study. We excluded the ”other” category from our analysis, as it did not provide meaningful information for our research objectives.The dataset’s primary complexities arise from inconsistencies in sampling rates across different sensors and significant instances of missing data, notably within the left wristwatch accelerometer data, which shows several intervals devoid of readings. Furthermore, the dataset’s recordings, spanning across the accelerometers and motion capture system, demonstrate varying sampling rates within the 30-second segments. Specifically, accelerometer data sampling rates fluctuate between 50 and 100 Hz, while the motion capture data is consistently sampled at around 100 Hz. These challenges underscore the need for robust processing techniques capable of handling asynchronous data and compensating for informational gaps within the dataset(Alia et al., 2021).

In our study, we utilize sensor data recorded from the arm, hip, and wrist, which serve as the primary data inputs for our model. To ensure uniformity across the dataset, we have interpolated all sensor readings to a consistent sampling rate of 100 Hz. Notably, we do not incorporate motion capture data into our analyses.Given the dataset’s challenges, including missing readings and inconsistent recording sessions, we have chosen to discretize activities within the same subject. This approach yields a variety of activity sequences, such as ”add, cut, open, making cereal” or ”cut, take, take, making cereal,” each representing different combinations of micro activities that lead up to a macro activity. The dataset presents variability not only in activity sequences but also in sensor availability per session, with some subjects lacking consistent sensor data across recordings. To address these discrepancies and the issue of missing data, we structured the dataset to ensure diversity in the training and testing sets, with distinct activity combinations and previously unseen activity types.It is important to note that the dataset provides only a general indication of atomic activities. Hence, most CHAR models that rely on time-specific atomic activity labels or sequential labeling are not suitable for this dataset. Our approach, tailored for this type of loosely labeled data, allows for robust activity recognition despite the absence of precise temporal annotations.In light of the inseparable nature of the segments within the dataset, we have preserved the original 30-second recording window size to maintain the integrity of the data. Consequently, the shape of our training set is (2326, 12, 3000), indicating that it comprises 2326 segments, each with 12 features, across 3000 time steps. Similarly, the testing set is structured as (711, 12, 3000), consisting of 711 such segments. This configuration ensures that the dataset’s original structure is retained, allowing for an authentic evaluation of our model’s performance.

To assess the precision of our model in detecting atomic activities, we utilize the Atomic Accuracy Score. This metric measures the proportion of atomic activities correctly identified above a specified confidence threshold $\alpha$ relative to the total number of activities detected. The score is mathematically defined as:

where $p_{i}$ denotes the confidence score of the $i$-th detected activity, $n$ is the total number of detected activities, and $\chi$ is the indicator function that returns 1 if $p_{i}>\alpha$, and 0 otherwise. In practical terms, we set $\alpha=0.4$, ensuring that only activities detected with a confidence level above 40% are considered in the accuracy measurement. This strategy focuses on the most reliably detected activities, thus providing a more meaningful assessment of the model’s performance in smart space environments.

In the evaluation of our model’s performance on complex activity classification, the F1 score is employed as a critical metric. The F1 score is a harmonic mean of precision and recall, providing a balanced measure that considers both the false positives and false negatives. This is especially important in our context where some complex activities may be underrepresented in the dataset.

The F1 score is calculated as follows:

In this study, we evaluate the performance of our proposed model against several established baselines in the field of CHAR. We provide a detailed description of our model in Subsection 3.5. For a fair comparison, we design the baselines with similar structures to ensure that each comparison model has approximately the same number of parameters. We employ the AdamW optimizer for training, with a maximum of 300 epochs. The following subsections detail the configurations of each comparison group:

• •

  ConvLSTM: Often considered a foundational architecture for addressing CHAR problems(Peng et al., 2018b; Varshney et al., 2022; Singh et al., 2023; Haresamudram et al., 2019), the ConvLSTM serves as a baseline to evaluate the enhancements our method brings to activity recognition. This model integrates a sensor fusion module utilizing CNN to extract primary sensor features, combined with LSTM networks to capture temporal dependencies among these features. The architecture culminates in a temporal convolution layer followed by a dense output layer, which collectively aim to optimize the recognition of complex human activities.

• •

  Concept Bottleneck(Koh et al., 2020): Utilizes a CNN-based structure to detect atomic concepts before advancing to higher-level concepts, applying MSE loss for concept identification. This hierarchical approach is possible to applied to complex activity recognition.

• •

  PEMM: The Pointwise Error Minimization Method (PEMM) is utilized as a comparative baseline in our study. It addresses potential concerns that Concept Bottleneck outcomes are limited by reliance on CNN architectures. To evaluate our model’s performance enhancements, we incorporate MSE in an ablation study, contrasting it with PEMM to highlight the effectiveness and advancements of our approach.

• •

  Debornair(Chen et al., 2021): This model parallels our basic ConvLSTM structure with a distinct preprocessing module that processes sensor data for atomic and complex activities separately before merging them. Debornair exclusively design for complex output only, focusing solely on the integration of processed data without predicting atomic activities.

• •

  XCHAR(Jeyakumar et al., 2023): Based on a vanilla ConvLSTM architecture, XCHAR differentiates itself by employing CTC(Connectionist Temporal Classification) loss to emphasize the importance of sequence in atomic activities. However, real-life datasets often suffer from errors or omissions in the sequencing of atomic activities, limiting the applicability of this model to certain datasets where precise sequence labeling is feasible.

In our study, we utilized a variance-based baseline to assess our model’s performance in recognizing complex activities. As shown in Table 4, our method surpasses other models with CHAR F1 Scores of 0.8463 on the Opportunity dataset, 0.8198 on the Cooking Challenge dataset, and 0.8657 on the FallAllD dataset, demonstrating its effectiveness in complex activity recognition. Nevertheless, the model encounters challenges in precisely detecting atomic activities when specific labels are available, recording a slightly lower Atomic Accuracy score of 0.6052, in comparison to XCHAR’s 0.6736 on the Opportunity dataset. While our approach mainly focuses in complex activity detection, further investigation into its capabilities for atomic activity recognition remains a point of interest.

Remarkably, our model demonstrates a significant advantage in environments where exact time labeling is absent, such as the Cooking dataset, where our method achieves the highest CHAR F1 Score of 0.8198 and Atomic Accuracy of 0.5209. This suggests that our variance approach adapatally handles datasets lacking detailed temporal annotations better than methods relying on point-wise error minimization like MSE loss, which performs well under conditions of precise labeling but struggles otherwise due to its inherent need for accuracy in individual assessments.The performance on the FallAIID dataset mirrors the trend observed in the Cooking dataset.

Moreover, the comparative performance in the table reveals that some models, including PEMM and Concept Bottleneck, compromise complex activity detection in favor of atomic activity recognition. For instance, while PEMM and Concept Bottleneck are designed to improve atomic detail, they fall short in overall CHAR F1 Scores compared to vanilla ConvLSTM, underscoring a trade-off (Gunning, 2016) that our method avoids. Our model’s multi-task recognition capability does not sacrifice the quality of complex activity detection, affirming its balanced approach in simultaneous task management.

The normalized confusion matrices illustrate distinct performance metrics across datasets. The FallAllD (fig 9) dataset demonstrates robust detection across all complex activities. For the Opportunity (fig 9) dataset, prediction scores for each complex activity exceed 0.8, with the ”early morning” activity achieving the highest accuracy at 0.97, whereas the ”clean up” activity records the lowest at 0.8. In contrast, the Cooking Challenge (fig 9) dataset reveals a suboptimal performance with the ”making cereal” activity scoring only 0.66, though other activities maintain scores above 0.8.

We assess the VCHAR, DeXAR, and Concept Bottleneck methods for complex activity recognition, focusing on their ability to represent results effectively within datasets characterized by sparse labeling.

• •

  DeXAR: (Arrotta et al., 2022) Initially designed for atomic activity recognition, we have modified the method to simultaneously estimate both atomic and complex activities. This adaptation incorporates an NLP-based visual representation to depict the model’s recognition results.In our example, as shown in Figure 11, the semantic visualization employs the Dexar encoding method to represent levels of confidence through color variations. Darker colors indicate higher confidence. This visualization illustrates the potential time intervals estimated by the model for each atomic activity, accompanied by textual explanations. However, it does not provide descriptions for complex activities.

• •

  Concept Bottleneck: (Koh et al., 2020) It merely indicates whether atomic activities are detected or not, without providing detailed information about the recognition results (fig 10). The output consists only of text descriptions, making it straightforward and concise. Nonetheless, this approach necessitates that users possess basic reading skills.In Figure 10, we observe that the complex activity is described solely through textual means. While some users may find this approach straightforward and intuitive, it essentially involves a sequence of actions—such as opening and closing a door, toggling a switch, and drinking—that collectively signify a complex activity, in this instance, ’making coffee’.

• •

  VCHAR: In Figure 13, the VCHAR system is illustrated, showcasing its capability to represent each atomic activity with a corresponding video and a descriptive label—in this example, ’making cereal’. The width of the video segment indicates the estimated time interval during which the activity occurs, as detected by VCHAR. Both the atomic and complex activities are labeled at the top of the graph. VCHAR is specifically designed to address issues of label sparsity. The time interval estimation is based on the weights connecting a specific last-layer neuron to all neurons in the time series layer, allowing for a comprehensive representation that integrates visual and textual descriptions for each activity.

As illustrated in Figure 13, VCHAR’s representation receives the highest median evaluation scores across the datasets—3.88 for Opportunity, 4.37 for Cooking Challenge, and 4.5 for Fallaid. These scores demonstrate a clear preference among users for more detailed and descriptive outputs to better understand complex scenarios, especially in everyday contexts. The distinct advantage of VCHAR is its ability to convey intricate sensor data interactions in a manner that is intuitive for laypersons. This preference underscores the importance of designing AI systems that not only perform well but also communicate their processes and results in ways that enhance user comprehension and trust in technology applications.

Another aspect of our evaluation focuses on illustrating to users how the model processes various data types to arrive at decisions. We assessed three distinct approaches: our proprietary method, a model-agnostic method, and a model-transparent method. This comparative study seeks to explore how different modeling approaches influence user preferences, particularly in terms of model interpretability and its impact on user satisfaction. Additionally, we analyze various explanation methods to identify which sensors are critical for recognizing atomic activities, further enhancing our understanding of each method’s effectiveness in practical scenarios.

• •

  Grad-CAM: (Lundberg and Lee, 2017) Grad-CAM is a model-transparent method that calculates the gradient of a target concept (output) relative to the feature maps of a designated layer. It produces a heatmap that identifies the critical sensors in higher layers that are pivotal for class prediction. As illustrated in Figure 14, it shows how various sensors contribute at different intensities to a particular activity, with brighter areas indicating greater importance.

• •

  SHAP (Chattopadhay et al., 2018) In contrast, SHAP is a model-agnostic method that approximates the relationship between inputs and outputs without probing the model’s internal mechanisms. It focuses on representing the sensor signals in the input time series data. SHAP calculates scores by estimating the impact of each feature on the prediction, using Shapley values from cooperative game theory to quantify each feature’s contribution.Shapley values in SHAP can be positive or negative, depending on whether a feature contributes positively or negatively to the model’s prediction for a particular instance, relative to the average prediction. This method highlights critical regions relative to the predicted class, as illustrated in Figure 15. Given the inherent complexity of sensor signals, simple descriptions of input may not fully capture their significance.

• •

  VCHAR VCHAR delivers a detailed depiction of sensor contributions by integrating both textual and visual representations. As shown in Figure 16, it visualizes a scenario where a person is attempting to open a fridge, with the most significant sensor activation value on the left foot, which VCHAR highlights in the visualization. Additionally, the sensor values are derived from gradient values tied to the selected activity type, akin to Grad-CAM’s methodology but extended to include both atomic and complex activities. VCHAR not only identifies these activities but also labels them in the visual representation, enhancing its analytical capabilities for detailed activity analysis.

From the results depicted in Figure 17, VCHAR demonstrates superior performance with median scores of 4.2, 3.8, and 4.5, along with a lower variance. Interestingly, our results are comparable to those of Grad-CAM. Grad-CAM scored lower than SHAP, which primarily stems from SHAP’s approach of explaining outcomes based on time-series data, thus providing a more detailed informational context. Despite this, our method, which also employs a model-transparent approach akin to Grad-CAM, received higher scores. This suggests that laypersons may benefit from more detailed explanations provided by our model, as compared to those used by expert users.