Evaluating Deep Networks for Detecting User Familiarity with VR from Hand Interactions

Virtual reality (VR) is increasingly being looked at as a mechanism for delivering experiences for tasks that users may typically perform in real-world, desktop, and mobile environments. VR environments are being evaluated across a diverse spectrum of users for education [1], therapy [2, 3], physical fitness [4], and even applications such as security [5, 6] and personal banking [7]. Research in behavior-based VR security shows that user actions in VR change at varying timescales [8, 9, 10]. As VR applications become more prevalent in consumer spaces, the level of first-time or early-stage VR users is expected to rise. For instance, an older adult may be recommended an exercise routine they can perform in a VR space. If the user is unfamiliar with the VR application, they may need timely intervention to provide training as otherwise they may become disincentivized and stop using the application. Though such training can be given by the service provider, automatic training delivery by detecting the familiarity of the user provides the benefit of providing on-demand training in the user’s personal environment, and alleviates the burden on strained service staff.

In this work, we provide a first attempt at conducting automated familiarity detection using the movements of a user as they engage in a VR task. We use a door opening VR task, where a user enters a 4-digit passcode combination to open a VR door, as users, regardless of prior VR experience, will have performed similar tasks in the real world. We envision VR doors to be a point of entry to collaborative virtual spaces. For example, a user in a virtual office setting may ‘walk’ to a common virtual conference room and enter a combination before entering the room. Early detection of prior VR experience, in this case through the interaction with the VR door, could enable real-time modifications to the interaction elements before the user enters the conference room to perform more complex interactions. In our approach, we track the finger movements of a user entering a passcode combination to unlock a VR door, and train deep neural networks to detect familiarity with VR. We use user-reported binary experience level with VR as a representation of familiarity. The task considered, i.e., unlocking a VR door, also has special significance in the area of VR security. Recognizing the potential for VR applications to store sensitive user data, a number of VR applications investigate leveraging the VR environment to enable secure access, through passcodes or biometric signatures [5, 6]. With the emergence of hand tracking for VR, recent work has explored user identification based on tracking hand data [11] In trying to gain access to a secured space in VR, a novice user may perform actions that cause them to be locked out by VR authentication mechanisms, e.g., pressing the wrong keypad button due to lack of knowledge on interaction process, or performing a deviating movement.

To the best of our knowledge, no work exists for learning-based detection of user familiarity with VR. Some work exists in an allied area of using machine learning for familiarity assessment for a particular task simulated in a VR environment, e.g., evaluating surgical skill. For instance, a considerable body of work exists to detect surgical skill, by tracking the eye or hand movements of the practitioner [12]. Work similarly exists in using eye movements to detect soccer player expertise in VR [13, 14]. Work has evaluated assessing spatial familiarity during wayfinding based on the eye movements performed by a user when turning at junctions [15]. However, the focus is on familiarity with the task, rather than with VR as an interaction medium itself. Work on understanding how different gaming controllers influence perceptions of usability have been explored [16] and provide insights on how changes in ergonomics can impact usage.

We create a VR environment where users interact with a numeric panel to unlock a VR door via hand tracking using a controller-free head-mounted device (HMD). We use the Meta Quest Pro in this study. As VR environments become pervasive, an access controlled door may be used to allow users with varying familiarity in VR to enter a virtual meeting room or their office. Users who are unfamiliar with VR would still be familiar with the task from the real world as most people have opened a door using a numeric panel. To unlock the door, the user uses their (controller-free) hands to enter a numeric passcode combination by interacting with VR buttons simulating those on a traditional access panel on a real-world door. Upon completing the entry, they press a key labeled ‘E’ to enter the combination and unlock the door. If the user enters the correct combination, the door opens immediately, and automatically closes and locks after 3 seconds. We design our panel’s buttons to consist of block game objects for digits 1 through 0, and the letters ‘E’ represent enter and ‘C’ representing clear. We use native hand tracking enabling the participant to use hand motions to interact with the game objects. We detect if a user has interacted with a button by detecting collisions using the collider on the tip of the index finger on both hands and the colliders on the buttons. Figure 1 shows a view of the door unlock application.

We recruited 14 participants from the student, faculty, and staff at Clarkson University. Prior to using the VR environment, we asked participants to indicate if they had prior experience in VR. Of the 14 participants, 7 reported no prior experience in VR, i.e., no familiarity, and the remaining 7 indicated being very familiar with VR. All participants in our study are right-handed with 10 participants self reporting as male and the remaining 4 as female. We collected data using a Meta Quest Pro and recorded position and orientation data at 60 frames per second. During data collection, we asked participants to enter $2648$, $2468$, $1379$, and $3179$ on the virtual keypad. Each combination was entered 10 times before moving to the next combination. Once the participant had entered a combination, they were asked to press ‘E’ to open the door. If the participant made a mistake, we asked them to press ‘C’ to clear the code and try again. Incorrect entries were stored, but not used during the training and testing phase. In Figure 2 we show all 10 trials for each door lock combination for one VR-familiar and one non-VR-familiar participant. We observe that the user who self reported as unfamiliar with VR shows a more variable pattern of movement.

We evaluate detection of VR familiarity by training classifiers on sliding windows extracted from the trajectory of the dominant hand of the participants in the dataset. We evaluate sliding windows of sizes 50, 60, 70, 80, 90, 100, 110, and 120, with a step size of 1. For each sliding window choice, we evaluate three types of classifiers—multi-layer perceptrons (MLPs), fully convolutional networks (FCNs) [17], and Point Cloud Transformer (PCT) [18]. We train one classifier per sliding window and per key combination.

Tables I and II summarize the results acquired by running the three classifiers examined in this work with various sliding window sizes and numeric passcode combinations. We show peak test accuracy in Table I and area under the ROC curve in Table II. Columns represent sliding windows, while rows represent classifiers and passcodes. We show the ROC curves for each of the models in Figure 3, with columns representing passcodes and rows representing classifiers. Each curve in a plot corresponds to a sliding window size.

As demonstrated by Table I, we obtain highest accuracy of 88.03% for the combination 2648 using the PCT classifier with sliding window size of 120. We see similar high scores for other classifiers with the 2648 combination. The performance drops for smaller window sizes, with the lowest accuracy of 69.79% obtained using an FCN. A reduction in the amount of information to learn contextual information may contribute to the drop in performance with reducing window sizes. The combination 3197 shows the next best set of accuracies, with the highest for the combination being 80.11% using the FCN. Similar to the 2648 combination, we see a drop in accuracy with reduced sliding windows. As seen in the plots in Figure 3, the FCN classifier shows higher overall area under the curve (AUC) for the 3197 combination compared to the MLP and PCT classifiers. PCT generally shows higher AUC for 2648. For lower window sizes, the FCN shows higher AUCs for both codes, and the MLP shows higher AUC for higher window sizes, potentially due to its simpler architecture.

We observe that 2468 and 1379 show results close to chance. A possible reason is that given the order of entry, i.e., 2648, 2468, and 1379, users may be getting acclimatized to the environment. With the 3197 combination, unlike the other 3, the user moves from right to left. During this motion, users not familiar with VR may need re-acclimatization time, though further data collection and analysis is necessary.

We demonstrate results of using deep networks to classify familiarity of a user in a VR environment using their movement patterns as they interact with a virtual keypad on a door. The task of opening a door after entering a combination is familiar to all users from having opened similar doors in the real world. We obtain highest accuracies upwards of 80% for combinations 2648 and next highest for 3197. Results indicate potential for the use of VR movements to detect familiarity, though the effect of time on acclimatization may need to be considered, i.e., conducting familiarity detection early is critical. As part of ongoing work, we are working on expanding our data collection to include a diverse array of users and VR tasks of varying complexity. With a broader dataset, we intend to expand prior familiarity from a binary label to a 5-point Likert scale. We are also interested in using eye and head movements to gauge whether distractions from the novelty of the environment play a role in familiarity detection, e.g., if a novice user spends time examining a new environment or is taken aback. As part of future work, we are interested in investigating whether on-demand training to familiarize novice users of VR is more effective than adapting the VR environment or controller to have a reduced interaction scope. The reduced interaction scope may be actualized by reducing the amount of movement in the VR environment when the person moves the analog stick, or adding pop-up hints when performing an interaction in the environment such as opening a door. Our current familiarity detection task in VR uses a door unlock panel which requires interactions with the dominant hand. Future studies on more complex tasks, or even unfamiliar tasks, that require both hands, or controllers, is of interest as most activities in VR involve a full body experience.