Showing 1–28 of 28 results for author: Paiva, A

Building Persuasive Robots with Social Power Strategies

Authors: Mojgan Hashemian, Marta Couto, Samuel Mascarenhas, Ana Paiva, Pedro A. Santos, Rui Prada

Abstract: Can social power endow social robots with the capacity to persuade? This paper represents our recent endeavor to design persuasive social robots. We have designed and run three different user studies to investigate the effectiveness of different bases of social power (inspired by French and Raven's theory) on peoples' compliance to the requests of social robots. The results show that robotic persu… ▽ More Can social power endow social robots with the capacity to persuade? This paper represents our recent endeavor to design persuasive social robots. We have designed and run three different user studies to investigate the effectiveness of different bases of social power (inspired by French and Raven's theory) on peoples' compliance to the requests of social robots. The results show that robotic persuaders that exert social power (specifically from expert, reward, and coercion bases) demonstrate increased ability to influence humans. The first study provides a positive answer and shows that under the same circumstances, people with different personalities prefer robots using a specific social power base. In addition, social rewards can be useful in persuading individuals. The second study suggests that by employing social power, social robots are capable of persuading people objectively to select a less desirable choice among others. Finally, the third study shows that the effect of power on persuasion does not decay over time and might strengthen under specific circumstances. Moreover, exerting stronger social power does not necessarily lead to higher persuasion. Overall, we argue that the results of these studies are relevant for designing human--robot-interaction scenarios especially the ones aiming at behavioral change. △ Less

Submitted 1 September, 2023; v1 submitted 12 July, 2023; originally announced July 2023.

ACM Class: I.2.9; H.5

Characterizing instance hardness in classification and regression problems

Authors: Gustavo P. Torquette, Victor S. Nunes, Pedro Y. A. Paiva, Lourenço B. C. Neto, Ana C. Lorena

Abstract: Some recent pieces of work in the Machine Learning (ML) literature have demonstrated the usefulness of assessing which observations are hardest to have their label predicted accurately. By identifying such instances, one may inspect whether they have any quality issues that should be addressed. Learning strategies based on the difficulty level of the observations can also be devised. This paper pr… ▽ More Some recent pieces of work in the Machine Learning (ML) literature have demonstrated the usefulness of assessing which observations are hardest to have their label predicted accurately. By identifying such instances, one may inspect whether they have any quality issues that should be addressed. Learning strategies based on the difficulty level of the observations can also be devised. This paper presents a set of meta-features that aim at characterizing which instances of a dataset are hardest to have their label predicted accurately and why they are so, aka instance hardness measures. Both classification and regression problems are considered. Synthetic datasets with different levels of complexity are built and analyzed. A Python package containing all implementations is also provided. △ Less

Submitted 4 December, 2022; originally announced December 2022.

From Modelling to Understanding Children's Behaviour in the Context of Robotics and Social Artificial Intelligence

Authors: Serge Thill, Vicky Charisi, Tony Belpaeme, Ana Paiva

Abstract: Understanding and modelling children's cognitive processes and their behaviour in the context of their interaction with robots and social artificial intelligence systems is a fundamental prerequisite for meaningful and effective robot interventions. However, children's development involve complex faculties such as exploration, creativity and curiosity which are challenging to model. Also, often ch… ▽ More Understanding and modelling children's cognitive processes and their behaviour in the context of their interaction with robots and social artificial intelligence systems is a fundamental prerequisite for meaningful and effective robot interventions. However, children's development involve complex faculties such as exploration, creativity and curiosity which are challenging to model. Also, often children express themselves in a playful way which is different from a typical adult behaviour. Different children also have different needs, and it remains a challenge in the current state of the art that those of neurodiverse children are under-addressed. With this workshop, we aim to promote a common ground among different disciplines such as developmental sciences, artificial intelligence and social robotics and discuss cutting-edge research in the area of user modelling and adaptive systems for children. △ Less

Submitted 20 October, 2022; originally announced October 2022.

Comments: Accepted proposal for a workshop to be held in conjunction with the 14th International Conference on Social Robotics (ICSR'22)

Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

Authors: Pedro P. Santos, Diogo S. Carvalho, Miguel Vasco, Alberto Sardinha, Pedro A. Santos, Ana Paiva, Francisco S. Melo

Abstract: We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully… ▽ More We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized), but the agents do not know beforehand which communication level they will encounter at execution time. To formalize our setting, we define a new class of multi-agent partially observable Markov decision processes (POMDPs) that we name hybrid-POMDPs, which explicitly model a communication process between the agents. We contribute MARO, an approach that makes use of an auto-regressive predictive model, trained in a centralized manner, to estimate missing agents' observations at execution time. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms relevant baselines, allowing agents to act with faulty communication while successfully exploiting shared information. △ Less

Submitted 5 June, 2023; v1 submitted 12 October, 2022; originally announced October 2022.

TensorAnalyzer: Identification of Urban Patterns in Big Cities using Non-Negative Tensor Factorization

Authors: Jaqueline Silveira, Germain García, Afonso Paiva, Marcelo Nery, Sergio Adorno, Luis Gustavo Nonato

Abstract: Extracting relevant urban patterns from multiple data sources can be difficult using classical clustering algorithms since we have to make a suitable setup of the hyperparameters of the algorithms and deal with outliers. It should be addressed correctly to help urban planners in the decision-making process for the further development of a big city. For instance, experts' main interest in criminolo… ▽ More Extracting relevant urban patterns from multiple data sources can be difficult using classical clustering algorithms since we have to make a suitable setup of the hyperparameters of the algorithms and deal with outliers. It should be addressed correctly to help urban planners in the decision-making process for the further development of a big city. For instance, experts' main interest in criminology is comprehending the relationship between crimes and the socio-economic characteristics at specific georeferenced locations. In addition, the classical clustering algorithms take little notice of the intricate spatial correlations in georeferenced data sources. This paper presents a new approach to detecting the most relevant urban patterns from multiple data sources based on tensor decomposition. Compared to classical methods, the proposed approach's performance is attested to validate the identified patterns' quality. The result indicates that the approach can effectively identify functional patterns to characterize the data set for further analysis in achieving good clustering quality. Furthermore, we developed a generic framework named TensorAnalyzer, where the effectiveness and usefulness of the proposed methodology are tested by a set of experiments and a real-world case study showing the relationship between the crime events around schools and students performance and other variables involved in the analysis. △ Less

Submitted 5 October, 2022; originally announced October 2022.

"Guess what I'm doing": Extending legibility to sequential decision tasks

Authors: Miguel Faria, Francisco S. Melo, Ana Paiva

Abstract: In this paper we investigate the notion of legibility in sequential decision tasks under uncertainty. Previous works that extend legibility to scenarios beyond robot motion either focus on deterministic settings or are computationally too expensive. Our proposed approach, dubbed PoL-MDP, is able to handle uncertainty while remaining computationally tractable. We establish the advantages of our app… ▽ More In this paper we investigate the notion of legibility in sequential decision tasks under uncertainty. Previous works that extend legibility to scenarios beyond robot motion either focus on deterministic settings or are computationally too expensive. Our proposed approach, dubbed PoL-MDP, is able to handle uncertainty while remaining computationally tractable. We establish the advantages of our approach against state-of-the-art approaches in several simulated scenarios of different complexity. We also showcase the use of our legible policies as demonstrations for an inverse reinforcement learning agent, establishing their superiority against the commonly used demonstrations based on the optimal policy. Finally, we assess the legibility of our computed policies through a user study where people are asked to infer the goal of a mobile robot following a legible policy by observing its actions. △ Less

Submitted 27 December, 2023; v1 submitted 19 September, 2022; originally announced September 2022.

Comparing Results of Thermographic Images Based Diagnosis for Breast Diseases

Authors: E. O. Rodrigues, A. Conci, T. B. Borchartt, A. C. Paiva, A. C. Silva, T. MacHenry

Abstract: This paper examines the potential contribution of infrared (IR) imaging in breast diseases detection. It compares obtained results using some algorithms for detection of malignant breast conditions such as Support Vector Machine (SVM) regarding the consistency of different approaches when applied to public data. Moreover, in order to avail the actual IR imaging's capability as a complement on clin… ▽ More This paper examines the potential contribution of infrared (IR) imaging in breast diseases detection. It compares obtained results using some algorithms for detection of malignant breast conditions such as Support Vector Machine (SVM) regarding the consistency of different approaches when applied to public data. Moreover, in order to avail the actual IR imaging's capability as a complement on clinical trials and to promote researches using high-resolution IR imaging we deemed the use of a public database revised by confidently trained breast physicians as essential. Only the static acquisition protocol is regarded in our work. We used lO2 IR single breast images from the Pro Engenharia (PROENG) public database (54 normal and 48 with some finding). These images were collected from Universidade Federal de Pernambuco (UFPE) University's Hospital. We employed the same features proposed by the authors of the work that presented the best results and achieved an accuracy of 61.7 % and Youden index of 0.24 using the Sequential Minimal Optimization (SMO) classifier. △ Less

Submitted 30 August, 2022; originally announced August 2022.

Journal ref: IWSSIP 2014 Proceedings

Modeling and mitigation of occupational safety risks in dynamic industrial environments

Authors: Ashutosh Tewari, Antonio R. Paiva

Abstract: Identifying and mitigating safety risks is paramount in a number of industries. In addition to guidelines and best practices, many industries already have safety management systems (SMSs) designed to monitor and reinforce good safety behaviors. The analytic capabilities to analyze the data acquired through such systems, however, are still lacking in terms of their ability to robustly quantify risk… ▽ More Identifying and mitigating safety risks is paramount in a number of industries. In addition to guidelines and best practices, many industries already have safety management systems (SMSs) designed to monitor and reinforce good safety behaviors. The analytic capabilities to analyze the data acquired through such systems, however, are still lacking in terms of their ability to robustly quantify risks posed by various occupational hazards. Moreover, best practices and modern SMSs are unable to account for dynamically evolving environments/behavioral characteristics commonly found in many industrial settings. This article proposes a method to address these issues by enabling continuous and quantitative assessment of safety risks in a data-driven manner. The backbone of our method is an intuitive hierarchical probabilistic model that explains sparse and noisy safety data collected by a typical SMS. A fully Bayesian approach is developed to calibrate this model from safety data in an online fashion. Thereafter, the calibrated model holds necessary information that serves to characterize risk posed by different safety hazards. Additionally, the proposed model can be leveraged for automated decision making, for instance solving resource allocation problems -- targeted towards risk mitigation -- that are often encountered in resource-constrained industrial environments. The methodology is rigorously validated on a simulated test-bed and its scalability is demonstrated on real data from large maintenance projects at a petrochemical plant. △ Less

Submitted 2 May, 2022; originally announced May 2022.

Avant-Satie! Using ERIK to encode task-relevant expressivity into the animation of autonomous social robots

Authors: Tiago Ribeiro, Ana Paiva

Abstract: ERIK is an expressive inverse kinematics technique that has been previously presented and evaluated both algorithmically and in a limited user-interaction scenario. It allows autonomous social robots to convey posture-based expressive information while gaze-tracking users. We have developed a new scenario aimed at further validating some of the unsupported claims from the previous scenario. Our ex… ▽ More ERIK is an expressive inverse kinematics technique that has been previously presented and evaluated both algorithmically and in a limited user-interaction scenario. It allows autonomous social robots to convey posture-based expressive information while gaze-tracking users. We have developed a new scenario aimed at further validating some of the unsupported claims from the previous scenario. Our experiment features a fully autonomous Adelino robot, and concludes that ERIK can be used to direct a user's choice of actions during execution of a given task, fully through its non-verbal expressive queues. △ Less

Submitted 2 March, 2022; originally announced March 2022.

MSC Class: 68T40 (Primary) 68U99 (Secondary) ACM Class: I.2.9; I.2.1; I.3.6; J.0

Geometric Multimodal Contrastive Representation Learning

Authors: Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S. Melo, Ana Paiva, Danica Kragic

Abstract: Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting… ▽ More Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting of modality-specific base encoders, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermediate representations to a latent representation space; ii) a multimodal contrastive loss function that encourages the geometric alignment of the learned representations. We experimentally demonstrate that GMC representations are semantically rich and achieve state-of-the-art performance with missing modality information on three different learning problems including prediction and reinforcement learning tasks. △ Less

Submitted 18 November, 2022; v1 submitted 7 February, 2022; originally announced February 2022.

Comments: ICML 2022 Camera ready version (update)

Projection-based Classification of Surfaces for 3D Human Mesh Sequence Retrieval

Authors: Emery Pierson, Juan-Carlos Alvarez Paiva, Mohamed Daoudi

Abstract: We analyze human poses and motion by introducing three sequences of easily calculated surface descriptors that are invariant under reparametrizations and Euclidean transformations. These descriptors are obtained by associating to each finitely-triangulated surface two functions on the unit sphere: for each unit vector u we compute the weighted area of the projection of the surface onto the plane o… ▽ More We analyze human poses and motion by introducing three sequences of easily calculated surface descriptors that are invariant under reparametrizations and Euclidean transformations. These descriptors are obtained by associating to each finitely-triangulated surface two functions on the unit sphere: for each unit vector u we compute the weighted area of the projection of the surface onto the plane orthogonal to u and the length of its projection onto the line spanned by u. The L2 norms and inner products of the projections of these functions onto the space of spherical harmonics of order k provide us with three sequences of Euclidean and reparametrization invariants of the surface. The use of these invariants reduces the comparison of 3D+time surface representations to the comparison of polygonal curves in R^n. The experimental results on the FAUST and CVSSP3D artificial datasets are promising. Moreover, a slight modification of our method yields good results on the noisy CVSSP3D real dataset. △ Less

Submitted 27 November, 2021; originally announced November 2021.

Journal ref: Computers & Graphics, 2 November 2021

How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agents

Authors: Miguel Vasco, Hang Yin, Francisco S. Melo, Ana Paiva

Abstract: This work addresses the problem of sensing the world: how to learn a multimodal representation of a reinforcement learning agent's environment that allows the execution of tasks under incomplete perceptual conditions. To address such problem, we argue for hierarchy in the design of representation models and contribute with a novel multimodal representation model, MUSE. The proposed model learns hi… ▽ More This work addresses the problem of sensing the world: how to learn a multimodal representation of a reinforcement learning agent's environment that allows the execution of tasks under incomplete perceptual conditions. To address such problem, we argue for hierarchy in the design of representation models and contribute with a novel multimodal representation model, MUSE. The proposed model learns hierarchical representations: low-level modality-specific representations, encoded from raw observation data, and a high-level multimodal representation, encoding joint-modality information to allow robust state estimation. We employ MUSE as the sensory representation model of deep reinforcement learning agents provided with multimodal observations in Atari games. We perform a comparative study over different designs of reinforcement learning agents, showing that MUSE allows agents to perform tasks under incomplete perceptual experience with minimal performance loss. Finally, we evaluate the performance of MUSE in literature-standard multimodal scenarios with higher number and more complex modalities, showing that it outperforms state-of-the-art multimodal variational autoencoders in single and cross-modality generation. △ Less

Submitted 31 January, 2022; v1 submitted 7 October, 2021; originally announced October 2021.

Comments: Accepted at the International Conference on Autonomous Agents and MultiAgent Systems (AAMAS) 2022

PyHard: a novel tool for generating hardness embeddings to support data-centric analysis

Authors: Pedro Yuri Arbs Paiva, Kate Smith-Miles, Maria Gabriela Valeriano, Ana Carolina Lorena

Abstract: For building successful Machine Learning (ML) systems, it is imperative to have high quality data and well tuned learning models. But how can one assess the quality of a given dataset? And how can the strengths and weaknesses of a model on a dataset be revealed? Our new tool PyHard employs a methodology known as Instance Space Analysis (ISA) to produce a hardness embedding of a dataset relating th… ▽ More For building successful Machine Learning (ML) systems, it is imperative to have high quality data and well tuned learning models. But how can one assess the quality of a given dataset? And how can the strengths and weaknesses of a model on a dataset be revealed? Our new tool PyHard employs a methodology known as Instance Space Analysis (ISA) to produce a hardness embedding of a dataset relating the predictive performance of multiple ML models to estimated instance hardness meta-features. This space is built so that observations are distributed linearly regarding how hard they are to classify. The user can visually interact with this embedding in multiple ways and obtain useful insights about data and algorithmic performance along the individual observations of the dataset. We show in a COVID prognosis dataset how this analysis supported the identification of pockets of hard observations that challenge ML models and are therefore worth closer inspection, and the delineation of regions of strengths and weaknesses of ML models. △ Less

Submitted 29 September, 2021; originally announced September 2021.

Assessing Machine Learning Approaches to Address IoT Sensor Drift

Authors: Haining Zheng, Antonio Paiva

Abstract: The proliferation of IoT sensors and their deployment in various industries and applications has brought about numerous analysis opportunities in this Big Data era. However, drift of those sensor measurements poses major challenges to automate data analysis and the ability to effectively train and deploy models on a continuous basis. In this paper we study and test several approaches from the lite… ▽ More The proliferation of IoT sensors and their deployment in various industries and applications has brought about numerous analysis opportunities in this Big Data era. However, drift of those sensor measurements poses major challenges to automate data analysis and the ability to effectively train and deploy models on a continuous basis. In this paper we study and test several approaches from the literature with regard to their ability to cope with and adapt to sensor drift under realistic conditions. Most of these approaches are recent and thus are representative of the current state-of-the-art. The testing was performed on a publicly available gas sensor dataset exhibiting drift over time. The results show substantial drops in sensing performance due to sensor drift in spite of the approaches. We then discuss several issues identified with current approaches and outline directions for future research to tackle them. △ Less

Submitted 2 September, 2021; originally announced September 2021.

Comments: 6 pages, The 4th International Workshop on Artificial Intelligence of Things, In conjunction with the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2021), Virtual conference, Aug. 14-18th

STFT-LDA: An Algorithm to Facilitate the Visual Analysis of Building Seismic Responses

Authors: Zhenge Zhao, Danilo Motta, Matthew Berger, Joshua A. Levine, Ismail B. Kuzucu, Robert B. Fleischman, Afonso Paiva, Carlos Scheidegger

Abstract: Civil engineers use numerical simulations of a building's responses to seismic forces to understand the nature of building failures, the limitations of building codes, and how to determine the latter to prevent the former. Such simulations generate large ensembles of multivariate, multiattribute time series. Comprehensive understanding of this data requires techniques that support the multivariate… ▽ More Civil engineers use numerical simulations of a building's responses to seismic forces to understand the nature of building failures, the limitations of building codes, and how to determine the latter to prevent the former. Such simulations generate large ensembles of multivariate, multiattribute time series. Comprehensive understanding of this data requires techniques that support the multivariate nature of the time series and can compare behaviors that are both periodic and non-periodic across multiple time scales and multiple time series themselves. In this paper, we present a novel technique to extract such patterns from time series generated from simulations of seismic responses. The core of our approach is the use of topic modeling, where topics correspond to interpretable and discriminative features of the earthquakes. We transform the raw time series data into a time series of topics, and use this visual summary to compare temporal patterns in earthquakes, query earthquakes via the topics across arbitrary time scales, and enable details on demand by linking the topic visualization with the original earthquake data. We show, through a surrogate task and an expert study, that this technique allows analysts to more easily identify recurring patterns in such time series. By integrating this technique in a prototype system, we show how it enables novel forms of visual interaction. △ Less

Submitted 1 September, 2021; originally announced September 2021.

Comments: 16 pages, 10 figures

FAtiMA Toolkit -- Toward an effective and accessible tool for the development of intelligent virtual agents and social robots

Authors: Samuel Mascarenhas, Manuel Guimarães, Pedro A. Santos, João Dias, Rui Prada, Ana Paiva

Abstract: More than a decade has passed since the development of FearNot!, an application designed to help children deal with bullying through role-playing with virtual characters. It was also the application that led to the creation of FAtiMA, an affective agent architecture for creating autonomous characters that can evoke empathic responses. In this paper, we describe FAtiMA Toolkit, a collection of open… ▽ More More than a decade has passed since the development of FearNot!, an application designed to help children deal with bullying through role-playing with virtual characters. It was also the application that led to the creation of FAtiMA, an affective agent architecture for creating autonomous characters that can evoke empathic responses. In this paper, we describe FAtiMA Toolkit, a collection of open-source tools that is designed to help researchers, game developers and roboticists incorporate a computational model of emotion and decision-making in their work. The toolkit was developed with the goal of making FAtiMA more accessible, easier to incorporate into different projects and more flexible in its capabilities for human-agent interaction, based upon the experience gathered over the years across different virtual environments and human-robot interaction scenarios. As a result, this work makes several different contributions to the field of Agent-Based Architectures. More precisely, FAtiMA Toolkit's library based design allows developers to easily integrate it with other frameworks, its meta-cognitive model affords different internal reasoners and affective components and its explicit dialogue structure gives control to the author even within highly complex scenarios. To demonstrate the use of FAtiMA Toolkit, several different use cases where the toolkit was successfully applied are described and discussed. △ Less

Submitted 4 March, 2021; originally announced March 2021.

Advancing from Predictive Maintenance to Intelligent Maintenance with AI and IIoT

Authors: Haining Zheng, Antonio R. Paiva, Chris S. Gurciullo

Abstract: As Artificial Intelligent (AI) technology advances and increasingly large amounts of data become readily available via various Industrial Internet of Things (IIoT) projects, we evaluate the state of the art of predictive maintenance approaches and propose our innovative framework to improve the current practice. The paper first reviews the evolution of reliability modelling technology in the past… ▽ More As Artificial Intelligent (AI) technology advances and increasingly large amounts of data become readily available via various Industrial Internet of Things (IIoT) projects, we evaluate the state of the art of predictive maintenance approaches and propose our innovative framework to improve the current practice. The paper first reviews the evolution of reliability modelling technology in the past 90 years and discusses major technologies developed in industry and academia. We then introduce the next generation maintenance framework - Intelligent Maintenance, and discuss its key components. This AI and IIoT based Intelligent Maintenance framework is composed of (1) latest machine learning algorithms including probabilistic reliability modelling with deep learning, (2) real-time data collection, transfer, and storage through wireless smart sensors, (3) Big Data technologies, (4) continuously integration and deployment of machine learning models, (5) mobile device and AR/VR applications for fast and better decision-making in the field. Particularly, we proposed a novel probabilistic deep learning reliability modelling approach and demonstrate it in the Turbofan Engine Degradation Dataset. △ Less

Submitted 1 September, 2020; originally announced September 2020.

Comments: The 3rd International Workshop on Artificial Intelligence of Things (AIoT) In conjunction with the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2020)

MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning

Authors: Miguel Vasco, Francisco S. Melo, Ana Paiva

Abstract: Humans are able to create rich representations of their external reality. Their internal representations allow for cross-modality inference, where available perceptions can induce the perceptual experience of missing input modalities. In this paper, we contribute the Multimodal Hierarchical Variational Auto-encoder (MHVAE), a hierarchical multimodal generative model for representation learning. In… ▽ More Humans are able to create rich representations of their external reality. Their internal representations allow for cross-modality inference, where available perceptions can induce the perceptual experience of missing input modalities. In this paper, we contribute the Multimodal Hierarchical Variational Auto-encoder (MHVAE), a hierarchical multimodal generative model for representation learning. Inspired by human cognitive models, the MHVAE is able to learn modality-specific distributions, of an arbitrary number of modalities, and a joint-modality distribution, responsible for cross-modality inference. We formally derive the model's evidence lower bound and propose a novel methodology to approximate the joint-modality posterior based on modality-specific representation dropout. We evaluate the MHVAE on standard multimodal datasets. Our model performs on par with other state-of-the-art generative models regarding joint-modality reconstruction from arbitrary input modalities and cross-modality inference. △ Less

Submitted 4 June, 2020; originally announced June 2020.

Explainable Agents Through Social Cues: A Review

Authors: Sebastian Wallkotter, Silvia Tulli, Ginevra Castellano, Ana Paiva, Mohamed Chetouani

Abstract: The issue of how to make embodied agents explainable has experienced a surge of interest over the last three years, and, there are many terms that refer to this concept, e.g., transparency or legibility. One reason for this high variance in terminology is the unique array of social cues that embodied agents can access in contrast to that accessed by non-embodied agents. Another reason is that diff… ▽ More The issue of how to make embodied agents explainable has experienced a surge of interest over the last three years, and, there are many terms that refer to this concept, e.g., transparency or legibility. One reason for this high variance in terminology is the unique array of social cues that embodied agents can access in contrast to that accessed by non-embodied agents. Another reason is that different authors use these terms in different ways. Hence, we review the existing literature on explainability and organize it by (1) providing an overview of existing definitions, (2) showing how explainability is implemented and how it exploits different social cues, and (3) showing how the impact of explainability is measured. Additionally, we present a list of open questions and challenges that highlight areas that require further investigation by the community. This provides the interested reader with an overview of the current state-of-the-art. △ Less

Submitted 18 February, 2021; v1 submitted 11 March, 2020; originally announced March 2020.

Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learning

Authors: Rui Silva, Miguel Vasco, Francisco S. Melo, Ana Paiva, Manuela Veloso

Abstract: In this work we explore the use of latent representations obtained from multiple input sensory modalities (such as images or sounds) in allowing an agent to learn and exploit policies over different subsets of input modalities. We propose a three-stage architecture that allows a reinforcement learning agent trained over a given sensory modality, to execute its task on a different sensory modality-… ▽ More In this work we explore the use of latent representations obtained from multiple input sensory modalities (such as images or sounds) in allowing an agent to learn and exploit policies over different subsets of input modalities. We propose a three-stage architecture that allows a reinforcement learning agent trained over a given sensory modality, to execute its task on a different sensory modality-for example, learning a visual policy over image inputs, and then execute such policy when only sound inputs are available. We show that the generalized policies achieve better out-of-the-box performance when compared to different baselines. Moreover, we show this holds in different OpenAI gym and video game environments, even when using different multimodal generative models and reinforcement learning algorithms. △ Less

Submitted 28 November, 2019; originally announced November 2019.

Fault Detection and Identification using Bayesian Recurrent Neural Networks

Authors: Weike Sun, Antonio R. C. Paiva, Peng Xu, Anantha Sundaram, Richard D. Braatz

Abstract: In processing and manufacturing industries, there has been a large push to produce higher quality products and ensure maximum efficiency of processes. This requires approaches to effectively detect and resolve disturbances to ensure optimal operations. While the control system can compensate for many types of disturbances, there are changes to the process which it still cannot handle adequately. I… ▽ More In processing and manufacturing industries, there has been a large push to produce higher quality products and ensure maximum efficiency of processes. This requires approaches to effectively detect and resolve disturbances to ensure optimal operations. While the control system can compensate for many types of disturbances, there are changes to the process which it still cannot handle adequately. It is therefore important to further develop monitoring systems to effectively detect and identify those faults such that they can be quickly resolved by operators. In this paper, a novel probabilistic fault detection and identification method is proposed which adopts a newly developed deep learning approach using Bayesian recurrent neural networks~(BRNNs) with variational dropout. The BRNN model is general and can model complex nonlinear dynamics. Moreover, compared to traditional statistic-based data-driven fault detection and identification methods, the proposed BRNN-based method yields uncertainty estimates which allow for simultaneous fault detection of chemical processes, direct fault identification, and fault propagation analysis. The outstanding performance of this method is demonstrated and contrasted to (dynamic) principal component analysis, which are widely applied in the industry, in the benchmark Tennessee Eastman process~(TEP) and a real chemical manufacturing dataset. △ Less

Submitted 26 June, 2020; v1 submitted 11 November, 2019; originally announced November 2019.

Comments: 43 pages, 23 figures. Accepted for publication in Computers & Chemical Engineering

MSC Class: 68T05

Expressive Inverse Kinematics Solving in Real-time for Virtual and Robotic Interactive Characters

Authors: Tiago Ribeiro, Ana Paiva

Abstract: With new advancements in interaction techniques, character animation also requires new methods, to support fields such as robotics, and VR/AR. Interactive characters in such fields are becoming driven by AI which opens up the possibility of non-linear and open-ended narratives that may even include interaction with the real, physical world. This paper presents and describes ERIK, an expressive inv… ▽ More With new advancements in interaction techniques, character animation also requires new methods, to support fields such as robotics, and VR/AR. Interactive characters in such fields are becoming driven by AI which opens up the possibility of non-linear and open-ended narratives that may even include interaction with the real, physical world. This paper presents and describes ERIK, an expressive inverse kinematics technique aimed at such applications. Our technique allows an arbitrary kinematic chain, such as an arm, snake, or robotic manipulator, to exhibit an expressive posture while aiming its end-point towards a given target orientation. The technique runs in interactive-time and does not require any pre-processing step such as e.g. training in machine learning techniques, in order to support new embodiments or new postures. That allows it to be integrated in an artist-friendly workflow, bringing artists closer to the development of such AI-driven expressive characters, by allowing them to use their typical animation tools of choice, and to properly pre-visualize the animation during design-time, even on a real robot. The full algorithmic specification is presented and described so that it can be implemented and used throughout the communities of the various fields we address. We demonstrate ERIK on different virtual kinematic structures, and also on a low-fidelity robot that was crafted using wood and hobby-grade servos, to show how well the technique performs even on a low-grade robot. Our evaluation shows how well the technique performs, i.e., how well the character is able to point at the target orientation, while minimally disrupting its target expressive posture, and respecting its mechanical rotation limits. △ Less

Submitted 18 November, 2019; v1 submitted 30 September, 2019; originally announced September 2019.

ACM Class: I.2.9; H.5.2; I.3.8; J.5

Software architecture for YOLO, a creativity-stimulating robot

Authors: Patrícia Alves-Oliveira, Samuel Gomes, Ankita Chandak, Patrícia Arriaga, Guy Hoffman, Ana Paiva

Abstract: YOLO is a social robot designed and developed to stimulate creativity in children through storytelling activities. Children use it as a character in their stories. This article details the artificial intelligence software developed for YOLO. The implemented software schedules through several Creativity Behaviors to find the ones that stimulate creativity more effectively. YOLO can choose between c… ▽ More YOLO is a social robot designed and developed to stimulate creativity in children through storytelling activities. Children use it as a character in their stories. This article details the artificial intelligence software developed for YOLO. The implemented software schedules through several Creativity Behaviors to find the ones that stimulate creativity more effectively. YOLO can choose between convergent and divergent thinking techniques, two important processes of creative thought. These techniques were developed based on the psychological theories of creativity development and on research from creativity experts who work with children. Additionally, this software allows the creation of Social Behaviors that enable the robot to behave as a believable character. On top of our framework, we built 3 main social behavior parameters: Exuberant, Aloof, and Harmonious. These behaviors are meant to ease immersive play and the process of character creation. The 3 social behaviors were based on psychological theories of personality and developed using children's input during co-design studies. Overall, this work presents an attempt to design, develop, and deploy social robots that nurture intrinsic human abilities, such as the ability to be creative. △ Less

Submitted 24 September, 2019; originally announced September 2019.

Comments: 17 pages, 7 figurs, 2 tables, open-source code, submitted to SoftwareX journal

Sharing and Learning Alloy on the Web

Authors: Nuno Macedo, Alcino Cunha, José Pereira, Renato Carvalho, Ricardo Silva, Ana C. R. Paiva, Miguel S. Ramalho, Daniel Silva

Abstract: We present Alloy4Fun, a web application that enables online editing and sharing of Alloy models and instances, to be used mainly in an educational context. By introducing the notion of secret paragraphs and commands in the models, it also allows the distribution and automatic evaluation of simple specification challenges, a useful mechanism that enables students to learn relational logic at their… ▽ More We present Alloy4Fun, a web application that enables online editing and sharing of Alloy models and instances, to be used mainly in an educational context. By introducing the notion of secret paragraphs and commands in the models, it also allows the distribution and automatic evaluation of simple specification challenges, a useful mechanism that enables students to learn relational logic at their own pace. Alloy4Fun stores all versions of shared and analyzed models, as well as derivation trees that depict how those models evolved over time: this wealth of information can be mined by researchers or tutors to identify, for example, learning breakdowns in the class or typical mistakes made by students and other Alloy users. A beta version of Alloy4Fun was already used in two formal methods courses, and we present some results of this preliminary evaluation. △ Less

Submitted 4 July, 2019; originally announced July 2019.

Nutty-based Robot Animation -- Principles and Practices

Authors: Tiago Ribeiro, Ana Paiva

Abstract: Robot animation is a new form of character animation that extends the traditional process by allowing the animated motion to become more interactive and adaptable during interaction with users in real-world settings. This paper reviews how this new type of character animation has evolved and been shaped from character animation principles and practices. We outline some new paradigms that aim at al… ▽ More Robot animation is a new form of character animation that extends the traditional process by allowing the animated motion to become more interactive and adaptable during interaction with users in real-world settings. This paper reviews how this new type of character animation has evolved and been shaped from character animation principles and practices. We outline some new paradigms that aim at allowing character animators to become robot animators, and to properly take part in the development of social robots. One such paradigm consists of the 12 principles of robot animation, which describes general concepts that both animators and robot developers should consider in order to properly understand each other. We also introduce the concept of Kinematronics, for specifying the controllable and programmable expressive abilities of robots, and the Nutty Workflow and Pipeline. The Nutty Pipeline introduces the concept of the Programmable Robot Animation Engine, which allows to generate, compose and blend various types of animation sources into a final, interaction-enabled motion that can be rendered on robots in real-time during real-world interactions. The Nutty Motion Filter is described and exemplified as a technique that allows an open-loop motion controller to apply physical limits to the motion while still allowing to tweak the shape and expressivity of the resulting motion. Additionally, we describe some types of tools that can be developed and integrated into Nutty-based workflows and pipelines, which allow animation artists to perform an integral part of the expressive behaviour development within social robots, and thus to evolve from standard (3D) character animators, towards a full-stack type of robot animators. △ Less

Submitted 3 June, 2019; v1 submitted 5 April, 2019; originally announced April 2019.

ACM Class: I.2.9; H.5.2; I.3.8; J.5

Learning multimodal representations for sample-efficient recognition of human actions

Authors: Miguel Vasco, Francisco S. Melo, David Martins de Matos, Ana Paiva, Tetsunari Inamura

Abstract: Humans interact in rich and diverse ways with the environment. However, the representation of such behavior by artificial agents is often limited. In this work we present \textit{motion concepts}, a novel multimodal representation of human actions in a household environment. A motion concept encompasses a probabilistic description of the kinematics of the action along with its contextual backgroun… ▽ More Humans interact in rich and diverse ways with the environment. However, the representation of such behavior by artificial agents is often limited. In this work we present \textit{motion concepts}, a novel multimodal representation of human actions in a household environment. A motion concept encompasses a probabilistic description of the kinematics of the action along with its contextual background, namely the location and the objects held during the performance. Furthermore, we present Online Motion Concept Learning (OMCL), a new algorithm which learns novel motion concepts from action demonstrations and recognizes previously learned motion concepts. The algorithm is evaluated on a virtual-reality household environment with the presence of a human avatar. OMCL outperforms standard motion recognition algorithms on an one-shot recognition task, attesting to its potential for sample-efficient recognition of human actions. △ Less

Submitted 6 March, 2019; originally announced March 2019.

Comments: 7 pages, 6 figures, submitted to 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Empathic Robot for Group Learning: A Field Study

Authors: Patricia Alves-Oliveira, Pedro Sequeira, Francisco S. Melo, Ginevra Castellano, Ana Paiva

Abstract: This work explores a group learning scenario with an autonomous empathic robot. We address two research questions: (1) Can an autonomous robot designed with empathic competencies foster collaborative learning in a group context? (2) Can an empathic robot sustain positive educational outcomes in long-term collaborative learning interactions with groups of students? To answer these questions, we dev… ▽ More This work explores a group learning scenario with an autonomous empathic robot. We address two research questions: (1) Can an autonomous robot designed with empathic competencies foster collaborative learning in a group context? (2) Can an empathic robot sustain positive educational outcomes in long-term collaborative learning interactions with groups of students? To answer these questions, we developed an autonomous robot with empathic competencies that is able to interact with a group of students in a learning activity about sustainable development. Two studies were conducted. The first study compares learning outcomes in children across 3 conditions: learning with an empathic robot; learning with a robot without empathic capabilities; and learning without a robot. The results show that the autonomous robot with empathy fosters meaningful discussions about sustainability, which is a learning outcome in sustainability education. The second study features groups of students who interact with the robot in a school classroom for two months. The long-term educational interaction did not seem to provide significant learning gains, although there was a change in game-actions to achieve more sustainability during game-play. This result reflects the need to perform more long-term research in the field of educational robots for group learning. △ Less

Submitted 5 February, 2019; originally announced February 2019.

Comments: ACM Transactions on Human-Robot Interaction, In press

Cognitive Architecture for Mutual Modelling

Authors: Alexis Jacq, Wafa Johal, Pierre Dillenbourg, Ana Paiva

Abstract: In social robotics, robots needs to be able to be understood by humans. Especially in collaborative tasks where they have to share mutual knowledge. For instance, in an educative scenario, learners share their knowledge and they must adapt their behaviour in order to make sure they are understood by others. Learners display behaviours in order to show their understanding and teachers adapt in orde… ▽ More In social robotics, robots needs to be able to be understood by humans. Especially in collaborative tasks where they have to share mutual knowledge. For instance, in an educative scenario, learners share their knowledge and they must adapt their behaviour in order to make sure they are understood by others. Learners display behaviours in order to show their understanding and teachers adapt in order to make sure that the learners' knowledge is the required one. This ability requires a model of their own mental states perceived by others: \textit{"has the human understood that I(robot) need this object for the task or should I explain it once again ?"} In this paper, we discuss the importance of a cognitive architecture enabling second-order Mutual Modelling for Human-Robot Interaction in educative contexts. △ Less

Submitted 22 February, 2016; originally announced February 2016.

Comments: Presented at "2nd Workshop on Cognitive Architectures for Social Human-Robot Interaction 2016 (arXiv:1602.01868)

Report number: CogArch4sHRI/2016/07