Socially Pertinent Robots in Gerontological Healthcare

To accomplish its objective of developing a socially pertinent robot, the SPRING consortium developed a novel software architecture for humanoid robots. The architecture is composed of eight modules (Figure 2) responsible for perception (self-localisation, human localisation, speech processing, human behaviour analysis, person manager) and action (experimenter interface, multi-party conversation, non-verbal behaviour generation) processes. They are developed on top of the ROS 1 Noetic²²2http://wiki.ros.org/noetic middleware. In total, all modules consist of 52 ROS nodes communicating through more than 170 ROS topics and actions. Where relevant, the architecture uses standard ROS messages. In particular, the SPRING architecture is one of the first to fully adopt the REP-155 ‘ROS4HRI’³³3https://www.ros.org/reps/rep-0155.html [22] standard for human-robot interaction to combine the different perception modalities (voices, faces, bodies) into a consistent representation of persons that can be used by downstream action modules.

The developed architecture is deployed on the PAL ARI humanoid robot, designed for use as a socially assistive companion [23]. ARI is 1.65m tall, with a differential-drive mobile base, a touch-screen on the torso, movable arms to gesture, and a head with LCD eyes that enable expressive gazing behaviours. It is equipped with a four-microphone array (front torso), an RGB camera (head), and two 180${}^{\circ}$ fish-eye cameras (chest and back) allowing us to capture and record the audio and video of the whole interaction from the robot’s perspective. An additional RGB-D camera is located in the front for self-localisation. The robot verbalises given responses using Acapela Text-To-Speech.⁴⁴4https://www.acapela-group.com/ The following sections provide details into the key features of each module.

The self-localisation module takes the RGB-D and Fisheye images as input and provides a 2D occupancy map indicating obstacles and the current position of ARI in the map. This information is crucial for obstacle avoidance during the navigation of ARI (Section 2.8). The developed module integrates RTABMap [24], a library for mapping and real-time tracking that incorporates ORB-SLAM [25], with an augmented version of the Hierarchical-Localisation (HLOC) [26] algorithm. RTABMap and HLOC provide two distinct, pre-aligned maps that share the same global coordinate system. The pre-alignment is done by aligning the camera centers registered in these maps. The module fuses both information to provide a single consistent map for other modules. This multi-layered approach is crucial for robust and precise localisation in complex and dynamic environments.

RTABMap uses the RGB-D camera that is slightly down-faced at the front of ARI to provide in many situations efficient mapping and localisation. However, certain environments might have reflective floors and RTABMap has trouble tracking key points, losing its localisation. For this purpose, an extended version of the HLOC algorithm was introduced to allow localisation based on the more robust and feature-rich images from the front and backwards-facing fisheye cameras. A calibration step, that allows to associated camera poses (position and orientation) and robot poses is done once-and-for-all at the beginning. Given a Fisheye image, the camera pose is initialised utilising the SENet [27] approach for advanced image retrieval. Image retrieval estimates the camera’s pose with an accuracy of a few meters, by identifying the most similar images from a collection of images of the environment with their poses in a global coordinate system. The next step is to establish a set of correspondences between the input image and the closest images. The culmination of this process is the pose estimation using the generalised absolute pose solver [28] from PoseLib⁵⁵5https://github.com/PoseLib/PoseLib, assuming known relative poses between the ARI cameras (i.e., camera rig) from external calibration.

Combining state-of-the-art image retrieval and pose estimation techniques, this complex methodology results in state-of-the-art localisation accuracy. Using front and backwards-facing images captured from two consecutive time points (i.e., four fisheye images), the system achieves localisation within the maps of the Broca gerontology day-care hospital with less than 1cm positional error and 0.2 degrees rotational error in 90% of queries. Comparatively, with a single image, the success rate drops to 15% for positional accuracy within 1cm and 30% for rotational accuracy within 0.2 degrees. These findings highlight the enhanced precision of the multiview approach over single-image HLOC localisation used to initialise the real-time camera pose tracking performed by ORB-SLAM.

The main goal of this module is to localise the humans in the space through time. To that aim, we use audio-visual data – more precisely the front fisheye images and the microphone array signals – to detect people, track them over time, and re-identify them (meaning that there is a one-to-one correspondence between persons and tracks). This information is exploited by the Conversation Manager (Section 2.7) to have a time-consistent identifier for each conversation partner and the human-aware navigation to move naturally around humans (Section 2.8). On the audio side, we employed a time-difference of arrival (TDOA) estimation for the audio modality, utilising an instantaneous version of the generalized cross-correlation phase-transform (GCC-PHAT) [29]. The GCC-PHAT algorithm reliably provides TDOA readings in frames with a single speaker. This implementation utilises two horizontal microphones from the Respeaker microphone array embedded in ARI. The TDOA readings are then translated into direction of arrival (DOA) estimates using geometric considerations On the visual side, we have implemented a state-of-the-art multi-person visual tracker known as fair multi-object tracking (FairMOT) [30]. This method combines the detection and the re-identification abilities and is based on the well-known residual neural network (ResNet34) [31]. However, the standard architectures are trained for regular cameras, while ARI’s camera has a fisheye lens. This required an annotation and retraining procedure described in [32]. Another important property of FairMOT is that the tracking is based on the Kalman filter, plus an additional matching step that associates detections (from the current time step) and Kalman predictions (from the previous time step), by means of a detection-to-prediction distance matrix. By computing the distance between DOA and the predictions of the Kalman filter, we can seamlessly incorporate audio detections to the tracking pipeline, thus achieving multi-modal speaker detection and tracking.

Based on the position of humans in the image, we can extract their pose using OpenPose [33]. This provides the orientation of each person and their feet position in the image. By triangulation and the assumption that persons stand on an even floor, we can recover their depth. This allows to infer the distance and orientation to each other to finally detect conversational groups (group centre and its members) using the Graph-Cuts for F-formation (GCFF) algorithm [34].

In real-life scenarios, a robot may engage with a group of people amid noisy and reverberant surroundings. The speech processing module’s objective is to generate multiple streams of transcribed speech for all speakers from the microphone array signals, which the conversational system (Section 2.7) will utilise. The transcribed text streams should maintain consistency over time. This may require various methods of attributing identity to the speakers, including DOA estimation and speaker identification.

We limit the scenarios to a maximum of two concurrent speakers. We also assume that the robot interacts with individuals in a half-duplex manner; namely, it does not listen while talking. The audio pipeline uses three steps to process the raw audio signal captured with the ReSpeaker microphone array:

1. 1.

   Enhance the speech quality;

2. 2.

   Extract key attributes (voice activity, DOA, speaker identity);

3. 3.

   Transcribe the enhanced audio data.

This process was designed to comprehensively handle the intricacies associated with single- and multi-speaker scenarios, using a systematic approach for audio processing and analysis.

This section describes the modules of ARI regarding the perception of human behaviour, namely: gaze target detection, detection of social acceptance of ARI, and multimodal emotion recognition. These modules input the position and groups of people from Section 2.3, the speech as processed in Section 2.4 as well as the face camera image, and could have direct applications in understanding the people’s intentions, emotional state, and engagement with the robot. While we did not have the opportunity yet to exploit the outcome of these modules in hospital settings, they are part of the software architecture and publicly available as the other ones.

The perception modules presented in Sections 2.3 (human localisation), 2.4 (speech processing) and 2.5 (human behaviour analysis) all extract what we call social features. To be used for downstream tasks (like the multi-party conversation manager, as described in the following section), these features have to be combined and associated with each other to build complete persons. The association might be direct (e.g., the facial recognition software module directly associates a face to a specific person), or indirect (we associate a body to a face based on the overlapping regions of interest in the source image, and transitively associate the body to a person). To broadcast possible associations between features and/or persons, with their corresponding likelihoods, the SPRING architecture uses the ROS4HRI [22] standard.

ROS4HRI defines five types of entities to model the humans interacting in the vicinity of the robot. The first three are feature entities for the face, body and voice. The last two are for persons and groups. Each entity has a unique identifier and properties, e.g. the bounding box of a face or the position of the group centre. The identifiers of feature entities are transients and might be created or changed at any time, based on the result of the face, body and voice detection algorithms. The person class, has instead a persistent identifier: a given person should always get assigned the same person identifier. Recognition modules are in charge of associating feature identifiers to the corresponding person identifier. For instance, the facial recognition node might broadcast a message like [{john, face_432, $0.8$}, {jane, face_432, $0.2$}] to indicate that a detected face has 80% chance of being John, and 20% chance of being Jane. This design allows the effective separation of concerns, where the question of ‘feature’ (face, body, voice) detection can be cleanly isolated from the question of feature matching. For SPRING, three feature matching processes are used: body to face, body to voice, and body to group. Associations depend in general on the closeness of two features, for instance, how far a detected face is from a body, and the Hungarian algorithm [55] is used to solve the assignment problem if several features have to be matched. Please note, due to ethical considerations during the experiments in the hospital, a direct association of a person entity to an actual person is not done (e.g. through a facial mapping with a database of photos). All person entities were regarded as anonymous persons.

This association mechanism is exploited to span, over time, a probabilistic graph of relationships between social features, persons and groups (Figure 3). The challenge is then to compute the most likely person–features associations. We have developed a novel algorithm, playfully named Mr. Potato algorithm [56], to compute the most probable associations between the different features and persons. Our algorithm searches all possible partitions of the graph, and selects the one that minimises the number of associations, while maximising affinity, i.e. the sum of likelihoods of each association. Our implementation represents efficiently the persons-features graph by the boost graph library [57]. Connected components are computed using a Depth-First-Search approach; likewise, minimum spanning trees are calculated using the Kruskal’s algorithm [58]; and shortest paths between nodes are computed using the Dijkstra algorithm [59]. The result of the algorithm is published as a new set of ROS4HRI-compatible topics, listing the list of tracked and known groups and persons, and their corresponding face, body and voice.

Commercial and research spoken dialogue systems (SDSs), conversational agents, and social robots have been designed with a focus on dyadic interactions. That is a two-party conversation between one individual user and a single system/robot. Such interactions can only be guaranteed in specific settings, like people interacting with Siri on their phone, or with Amazon Alexa in single-occupant homes. When Alexa is in a family home, its lack of multi-party capabilities is apparent [60], but this becomes a critical limitation when deploying social robots in public spaces [2, 3, 4, 5, 6, 7], where multi-party conversations (MPCs), involving people talking to both the robot and each other, do commonly occur.

Tasks that are typically trivial in the dyadic setting become considerably more complex when conversing with multiple users [61, 62]: (1) The speaker is no longer simply the other person, so the meaning of the dialogue depends on recognising who said each utterance (see (A) in Figure 4); (2) addressee recognition is similarly more complicated as people can address each other, the robot, and groups of individuals; and (3) response generation depends on who said what to whom, relying on the semantic content and surrounding multi-party context. To make things even more difficult, MPCs provide additional unique challenges that are underexplored. Dyadic SDSs must identify and answer the user’s goals to be practically useful. In MPCs, users can provide another person’s goal (see (B) in Figure 4), answer each other’s goals (see (C) in Figure 4), and even share goals (see (D) in Figure 4, [63]). In SPRING, we have established the task of multi-party goal-tracking [53].

The conversational system [64, 21] has been iteratively improved through regular user tests and interviews with patients visiting the Broca gerontology day-care hospital. The initial system [7] was developed before recent LLM advances (such as ChatGPT), relying on a ‘traditional’ modular architecture based upon Alana V2 [65, 66]. As patients were usually accompanied by a companion, the lack of multi-party capabilities proved problematic. The system interrupted users since it responded to every turn, not allowing them to talk to each other at any point. We therefore designed and ran a multi-party data collection in a wizard-of-oz setup [67, 53] (see Section 3) and have used this data to motivate and evaluate our current SDS. Not only is this new system multi-party and multimodal, it improves QA accuracy, improves accessibility for people with dementia [21], and enables added functionality. For example, where previously we had to specifically design the system to tell jokes and run entertaining quizzes [68, 54], LLMs can now handle this inherently due to their general knowledge. Most importantly, both users and the hospital staff have reported that the user experience has improved dramatically (see Section 3).

Gaze, gestures, and navigation of ARI are controlled by two modules: The behaviour manager interfaces a high-level planner with the conversational system to choose appropriate actions during an interaction. The behaviour generation module provides and executes the actions.

During the experimentation, the robot will not be accompanied by computer science researchers or engineers, but by medical researchers and personnel. In order to conduct the experimental session, an appropriate experimenter-interface was developed. Through a conception-trial-update process, we converged on an interface design that is a trade-off between usability and controllability. The interface (Figure 6) is implemented for use on a dedicated tablet.

From a technical point of view, the interface connects to its server that runs on the external computer. This server communicates with the modules described above via ROS, interacts with the tablet via web technologies, and works as a gateway between these two parts. Additionally, it allows the experimenter to control the data collection. Via a web browser and the tactile screen, the experimenter can easily check the status of the robot and external computer, conduct the experiment, control the interaction as well as generate quick annotations of the interaction that are synchronised with the rest of the ROS messages. These annotations enable the project engineers to measure error rates, and diagnose bugs more easily. More details about the experimenter interface and other user interfaces (e.g. for data collection) can be found in [78].

In this study, the user experience was assessed by two standardised scales. The first quantitative measure is the acceptability of the robot assessed with the Acceptability E-scale (AES) [81]. It is designed to measure the subjective acceptability of a system using 6 items, resulting in a global score that ranges from 6 to 30. For consumer ready-to-use products, the acceptability cut-off score is 25/30. The second quantitative measure is the usability of the robot assessed with the System Usability Scale (SUS) [82]. It is a 10-items scale designed to assess the overall user-friendliness of a system, and generating an overall score out of 100, where a higher score indicates better user-friendliness. For consumer ready-to-use products, the usability cut-off score is 72/100. Statistical differences were assessed using independent Student t-tests

The participant recruitment process began two weeks before the day of the experiment. A researcher used the management software of the day-care hospital [83] to identify patients who met the eligibility criteria. Once this information was validated, the researcher contacted the participant by telephone to present the objectives of the study and to arrange a time slot on the day of their medical visit to the hospital if the participant was willing to accept to take part in the tests. The researcher sent afterwards the information letter detailing the study by postal mail, together with a reminder letter of the appointment.

On the day of the test, the researcher checked that the tools (robot, tablet, camera) were working properly. Once the participants arrived, the researcher reminded them of the objectives of the study and the evaluation procedures. Participants could be escorted by one accompanying person. In case they confirmed their agreement to take part in the experiment, they were given a consent form to sign. The participant(s) were then invited to stand in front of the robot to begin the interaction with it. Two researchers led the activity, a facilitator, and a person who controlled the tablet and completed the observation grid. The facilitator’s role was to direct the test session and provide assistance or additional information to participants when needed.

Two separate waves of experimentation were conducted during this study, depending on the status of the architecture. The First wave (May 2023 – July 2023) used the initial version of the robot dialogue module, with an off-the-shelf automatic speech recognition model, and with almost no multi-party capabilities. In the Second wave (Sept. 2023 – Jan. 2024), some improvements to the system were made: The dialogue management included large-language models, the ASR was fine-tuned, the experimenter interface was updated, all-in-all to improve the user experience. The improvements were motivated by the participants’ feedback from the First wave.

The recruitment efforts previous to the two waves lead to two sets of participants with slightly different age profiles for the patients, and wider age differences for the companions (see Table 1). The study counts a total of 63 participants with an overall average age of 74.5 (±12.31) years.

We also report the average and standard deviation of the AES and SUS scores (Table 2), split by wave and user group, as well as the overall score per wave.

We also asked them what “use-case” they think would be more useful (Figure 7), among the following choices: (i) reception and welcoming, (ii) promoting social interaction without health risks, (iii) help in preparing for consultations, (iv) orientation and guidance, and (v) entertainment.

By looking at the main results, when we report the average AES and SUS results (Table 2), we observe a clear improvement between the First and Second wave, independently of the metric and for both the patient and companion group. A significant difference was found for the AES ($t(20)$ $=$ $-2.991$, $p$ $<$ $.05$) as the scores increased between the first ($M$ $=$ $15.5$, SD $=$ $5.88$) and the second wave ($M$ $=$ $20.8$, SD $=$ $5.20$). This significant difference was only found in patients ($t(18)$ $=$ $-2.906$, $p$ $<$ $.05$) and not in companions. However, no significant differences were observed for the SUS between the first and the second wave of experiments, both in patients and in companions. While there is an overall clear trend of improvement, the trend for the companion group is less pronounced than for the patient group.

We explain this improvement in acceptability and usability by the technical improvements we made to the software architecture (since the robot’s appearance did not change between waves): namely, the fine-tuning of the ASR module and the inclusion of LLMs in the conversation manager. These two major modifications, together with bug fixes, allowed a more natural interaction with the users since (i) ARI understood better what the participants were saying and (ii) ARI was able to answer to a wider range of questions with reasonable (although now always exact) answers. In particular, the participant’s feedback reported: enjoyment in using the robot, usefulness, acceptability of the robot’s reaction time (talking) and overall satisfaction with the robot.

It is also interesting to observe, when considering the two groups separately, that the companions tend to provide more positive AES and SUS feedback than the patients. For the time being, we have not identified a reason that justifies this difference.

One of the major challenges we encountered during our project concerned the recruitment of patients for the tests. These challenges can be attributed to a number of factors, including organisational aspects, last-minute cancellations and refusal to interact with the robot.

Some factors had a significant impact on the organisation of the experiments, for example, patients that refused to take part in the experiments, and coordination difficulties among the various hospital stakeholders (patients arriving late, consultations’ delay, or some experiments lasting longer than planned resulting in a delay in the following appointments). Finally, some participants refused to interact with the ARI robot during the first physical encounter, which directly compromised the interaction and made the experiment impossible. The reasons for this refusal were diverse, ranging from technological apprehension linked to the size of the robot, and ethical concerns.

During the First wave, some participants felt frustration specifically due to the limitations in terms of conversational capabilities, whether these were related to the performance of the ASR or the topic restrictions of the dialogue manager. Since these were identified as the two main blocking points, we paid attention to enhancing these specific skills, leading to a clear improvement in ARI’s acceptability and usability in the Second wave.