GameVibe: A Multimodal Affective Game Corpus

Affective Computing (AC) is the interdisciplinary field that refers to the study of human emotions and the development of tools and technologies that learn, interpret and perceive these emotions [1]. The availability of large-scale corpora comprising affect manifestations elicited through appropriate stimuli is critical for AC. However, manifestations of affect are often highly subjective and difficult to assess; each individual’s interpretation of a stimulus is influenced by factors such as preferences, memory, biases, systematic errors, and expectations [2, 3]. Furthermore, emotions are a dynamic phenomenon and participants’ reactions to similar stimuli may shift over time. Therefore, as AC advances and deep learning methods require an increasing amount of data and scale, it becomes important to design and implement experimental protocols that maximise the reliability of collected labels of large-scale affect corpora.

Video games are a hugely popular type of digital media, offering a unique form of human-computer-interaction (HCI) due to their rich content and interactive nature [4]. Naturally, the global video game industry has seen substantial expansion and is already among the fastest growing subsectors within the broader entertainment sector [5]. Thus, using games as a test-bed to reveal the intricacies of the HCI loop is one of the most promising ways to analyse human behaviour and experience, at scale. Studying the behaviour of players and the experience of play can, in turn, be used to improve the technical aspects of game development, and to contribute to more engaging and personalized game experiences [6, 7].

The literature already contains a growing number of diverse affect corpora that vary in terms of modalities of user input considered, annotation methods, affect stimuli, and number of participants among other factors. Some notable examples of modalities recorded include audiovisual data [8, 9], physiological signals [10], and facial expression data [11]. First-person annotation protocols ask the annotator to label their own experience, and have been employed in several affect corpora, including MazeBall [12], PED [13], FUNii [14], and MUMBAI [9]. When employing a third-person protocol, instead, we ask annotators to label the experience of another person, such as in the RECOLA [15], LIRIS-ACCEDE [16], AFF-Wild [10], AffectNet [11], and SEWA [17] corpora. The provided labels can be discrete (e.g. categories, scales) such as in GAME-ON [8] and BIRAFFE2 [18] or continuous traces as in RAGA [19] and MUMBAI [9].

Contemporary affect corpora are gradually deviating from the controlled setting of in-lab experiments [15, 20, 21] to real-life (or in-the-wild) scenarios [18, 22, 11] in an attempt to elicit more natural user behaviours and manifestations of affect. Hiring annotators from crowdsourcing platforms has also been gaining interest in an attempt for AC to scale. Untrained crowd workers in uncontrolled settings, however, can be unreliable, resulting in corpora of questionable, or even limited, value [23]. It is fair to say that all affect corpora are characterized by subjectivity [24, 25, 26], since consensus among annotators is not necessarily required. Whilst some studies attempt to mitigate this issue of variability (e.g. by using mood induction [27]) for better performance in downstream tasks, doing so may hamper the generalizability and reliability of results when deployed in uncontrolled real-life scenarios. Motivated by these issues, we seek to provide an affect corpus with validated annotator reliability and sufficient diversity in the stimuli to promote generalizability in downstream tasks. We aim to accomplish this by following a tried and tested protocol for quality assurance [23], and collect stimuli which present rich contextual variety within a single domain to promote deeper research into the generalizability of affect predictions.

Regarding games as affect stimuli, the players’ (or viewers’) perceptions of a gameplay video as stimulus are inextricably linked to the game genre, the form of interfacing, the game’s objective, the number of players, and potential social aspects [4]. We specifically focus on the First-Person Shooter (FPS) genre of games for this paper, for many reasons. FPS games have been popular among players since the 1990s. Moreover, FPS games usually have high-quality graphics and audio, and a rich variability of stimuli due to the vast number of FPS games developed over the years. FPS games tend to be highly stimulating for both players and viewers, and share many fundamental gameplay elements whilst simultaneously employing very different styles (in terms of both art and gameplay). In addition, the FPS genre has been hugely popular on live-streaming platforms such as Twitch for several years. Indicatively, during December 2023, 4 of the top 10 live-streamed categories were FPS games, with over 220 million viewer hours watched in just 30 days¹¹1Data taken from https://www.twitchmetrics.net/games/viewership, accessed on 16/12/2023.. We argue that collectively these factors make FPS games the ideal genre for studying HCI, especially for understanding viewer engagement. This makes research in FPS affect modelling highly useful for content creators, streamers, and game designers.

With this in mind, we introduce GameVibe, a novel multimodal affect corpus of viewer engagement for FPS game videos. This corpus consists of 2 hours of high-quality audio and visual data from 30 different FPS games extracted and curated from publicly available “Let’s Play” videos on YouTube. Data collection involved 20 annotators, consisting of trained researchers, as well as postgraduate and undergraduate students. Affect labels were provided in the form of unbounded, time continuous signals using the RankTrace annotation tool [6] on the PAGAN platform [28]. As part of the corpus, we provide the raw videos used as stimuli during data collection, as well as latents extracted using pre-trained foundation models for visuals (VideoMAE [29] and MVD [30]), and audio (BEATS [31]) for use in downstream tasks. Furthermore, we include quality assurance data on each annotator to give a better understanding of the reliability and validity of the labels in the dataset. Finally, we test the reliability and validity of the dataset in terms of inter-annotator agreement. We believe that the rich variety in GameVibe’s stimuli, which encompasses multiple game modes, winning conditions, and art styles, presents a unique opportunity for research on the ability of affect models to generalise across a wide variety of contexts within a single genre, such as FPS games.

Measuring the experience of digital game enjoyment and accurately identifying which game elements engage players are important goals for both the study of user experience in games and the development of better games [4]. In this section, we introduce the FPS affect corpus GameVibe, which was solicited to provide a rich, multimodal dataset of elicitors for viewer engagement. The collected dataset includes: (a) synchronised frames and audio per game, (b) extracted latent representations from audiovisual data, (c) annotation traces per participant (in raw & processed form), (d) participants’ replies to demographic surveys (anonymised data). The novelty of the dataset lies in the fact that affect annotations are user specific, thus we exploit the different opinions and perception of the users for the game that can enable the training of fair affect models that are highly robust to unseen data.

In this section, we provide a detailed description of the process we followed to build GameVibe. The core phases, as illustrated in Fig. 1, are as follows:

• •

  Design phase: we outline the objectives, tools, methodologies, and parameters of the study.

• •

  Stimuli collection phase: we choose and curate video content from diverse FPS games.

• •

  Annotation phase: before collecting labels of viewers’ emotional states, we recruit participants, procure and prepare the equipment, and develop informed consent forms to uphold ethical standards. In addition, we conduct Quality Assurance tests to measure participants’ reliability. We detail all the above below.

• •

  Post-experiment phase: we assess the quality of the dataset, process and analyse the collected data and document our findings. This ensures the dataset’s efficacy and utility for subsequent research endeavours.

The dataset is split into five directories as depicted in Fig. 3. The “Stimuli” directory contains the 1-minute game videos organised into subdirectories by session and named according to the game in each video. The “Questionnaire” directory contains participants’ responses to the post-experiment questionnaire in CSV format, and a Python script to process the responses and create visualisations. The “Annotations” folder contains the annotation data returned from PAGAN and the Python scripts required to process them; this folder has several subdirectories of importance. In the “Raw” subdirectory, the raw annotations outputted by PAGAN for each session can be found. The “Raw Combined” subdirectory contains the raw data assembled into a single CSV for each task, and the ground truth signals for the QA test (see Methods). The “Processed” subdirectory contains the processed annotation data for the QA tasks and the Engagement task for all sessions in NumPy file format. These processed files consist of a Python dictionary containing the following hierarchy: data is sorted into sessions, which in turn are sorted into participants, and further sorted by game where the participants’ annotations can be found. The “Latent Extraction” subdirectory contains Python scripts to extract the latents from the video and audio found in the “Stimuli” directory. We also include examples of extracted latents for a 3-second time window that can be used for further analysis. Finally, the “Analysis” directory contains the scripts used to conduct the analysis in this paper, and includes a “Processed” subdirectory with annotation data with outliers removed (see Methods).

Annotator Quality Assurance: We first illustrate the reliability of our annotators by computing their average QA scores across our visual and auditory QA tasks (see Methods). Since both QA tasks involve an annotation task where the ground truth (screen brightness and audio frequency respectively) is known in advance, we use the Signed Differential Agreement (SDA) metric [36] to measure similarity. We chose SDA as it has been proven effective in our previous QA study [23] and, due to its bounded nature between -1 and 1, is an intuitive metric for selecting a filter threshold. Across all participants and both QA tasks, the average SDA score was $0.43\pm 0.15$, meaning that annotators correctly annotated $71.5\%$ of all time windows among QA signals. If we used a QA filtering threshold value similar to previous work[23] (SDA = 0, or $50\%$ accuracy), only 3 annotators (P6 and P7 from Session 2, P17 from Session 4) would be removed due to poor QA performance. This verifies that almost all our annotators understood the annotation process and could accurately produce labels before moving on to the engagement task. For the purposes of the technical validation of this paper, however, no participants are removed through this QA filtering in order to assess all raw data included in the dataset.

Inter-Annotator Agreement: We use DTW (see Methods) to create distance matrices for each video in each session, in order to calculate the inter-annotator agreement between annotators’ normalized engagement traces on the same stimuli. We average those DTW values between pairs of participants on a per-session basis, since the same participant with the same identifier (e.g. P1) annotated every video in the same session. We report the average DTW distance across all stimuli (30) per session in Table 1, using different participant identifiers because different participants annotated different sessions. We observe that there are differences between annotators; only in Session 1 annotators were more in agreement. However, annotators are rarely consistently disagreeing with all other annotators; the most obvious instances of this are P6 (with DTW distances over 10 with all other annotators in Session 2) and P17 (with DTW distances over 11 with three other annotators in Session 4). We note that both annotators performed poorly in the QA tasks as well, and would have been removed via an SDA cut-off on the two QA tasks (see above). In general, there is a statistically significant ($p<0.05$) negative Pearson correlation between average DTW distance of each participant’s engagement traces with all others in the same session and SDA score averaged from the two QA tasks ($r=-0.56$). This is a promising finding for the quality assurance offered by the additional QA tasks, as the closer the annotators are to the known ground truth in those two tasks (an SDA score near 1), the less likely they are to disagree with other annotators in the more challenging engagement annotation task.

Outliers per participant: We detect outliers on the engagement traces (see Methods) based on the KDE of all DTW matrices of all videos in the same session, and report results with two filters for outlier removal. When using the $90\%$ filter (see Methods), 60 out of 600 annotations are removed (15 per session), whilst with the $80\%$ filter 120 annotations are removed (30 per session). Interestingly, while P7 would be removed due to poor QA task performance, none of their 30 traces are removed as outliers for the 90% filter and only 2 traces are removed as outliers for the 80% filter. On the other hand, P17 has the most outliers of all 20 participants: 11 of their 30 traces are removed as outliers for the 90% filter and an extraordinary 16 out of 30 removed for the 80% filter. Other participants with many removed outliers are P4 (with 9 outliers for 80% filter) and P3, P13, P20 (each with 8 outliers for 80% filter). While average DTW distance was correlated with the average SDA score from the two QA tasks, the same metric is not significantly correlated ($p>0.05$) with the number of outliers removed: $r=-0.33$ for removed outliers per participant with the 90% filter and $r=-0.25$ for outliers with the 80% filter.

Outliers per game and video: It is worthwhile to observe which game stimuli were more prone to inter-annotator disagreements, which would lead to more outliers removed through the above filtering process. After applying the $90\%$ filter, outliers were removed from 26 out of the 30 games. The games with no outliers were Apex Legends, Wolfram, Medal of Honour (PS1) and Superhot. The games with the most outliers were Outlaws, Counter-Strike 16, and Wolfenstein 3D with 4 outliers each (out of 20 annotation traces across 4 videos of the same game). After applying the $80\%$ filter, outliers were removed from every game except Apex Legends. The game with the most outliers when using the $80\%$ filter was Operation Body Count, with 9 outliers removed (out of 20 traces). The consistency of Apex Legends across sessions and annotators is likely due to intertwined factors, such as its highly stylised graphics, strong emphasis on audio cues, and quick well-defined changes in tempo (e.g. alternating between fast-paced team fights and slow-paced periods of healing, looting, exploration, etc.). Finally, observing individual videos with the most outliers after applying the $90\%$ filter, Session 1’s Outlaws and Session 4’s Counter-Strike 16 had 3 outliers removed each (out of 5 traces each). When using the stricter $80\%$ filter, Session 4’s Superhot video had all of its annotations classified as outliers and removed, indicating that this video had particularly poor inter-rater agreement compared to others in that session. This aggressive filtering process could evidently prove problematic, as in the case of this video, no data remain for affect modelling. For this reason, the data records contain all raw and processed traces and Python scripts that would allow researchers to perform their own filtering process or apply different KDE thresholds depending on their goals.

We provide annotation data in CSV format and the respective video stimuli in .mp4 format, both formats can be processed in most software packages or programming language. We provide Python files (.py) for data processing and extraction of feature representations are provided. The GameVibe_README.md file details the organisation of the dataset, explaining the structure, naming convention, and specific contents of each file.

The dataset can be managed, visualised and pre-processed using Python files (.py) files. These files are accessible for download at: https://osf.io/p4ngx/?view_only=fc9c68cf3f104ad7afba5ab73a1c66a8.