Showing 1–13 of 13 results for author: Doyle, P R

Cross-Cultural Validation of Partner Models for Voice User Interfaces

Authors: Katie Seaborn, Iona Gessinger, Suzuka Yoshida, Benjamin R. Cowan, Philip R. Doyle

Abstract: Recent research has begun to assess people's perceptions of voice user interfaces (VUIs) as dialogue partners, termed partner models. Current self-report measures are only available in English, limiting research to English-speaking users. To improve the diversity of user samples and contexts that inform partner modelling research, we translated, localized, and evaluated the Partner Modelling Quest… ▽ More Recent research has begun to assess people's perceptions of voice user interfaces (VUIs) as dialogue partners, termed partner models. Current self-report measures are only available in English, limiting research to English-speaking users. To improve the diversity of user samples and contexts that inform partner modelling research, we translated, localized, and evaluated the Partner Modelling Questionnaire (PMQ) for non-English speaking Western (German, n=185) and East Asian (Japanese, n=198) cohorts where VUI use is popular. Through confirmatory factor analysis (CFA), we find that the scale produces equivalent levels of goodness-to-fit for both our German and Japanese translations, confirming its cross-cultural validity. Still, the structure of the communicative flexibility factor did not replicate directly across Western and East Asian cohorts. We discuss how our translations can open up critical research on cultural similarities and differences in partner model use and design, whilst highlighting the challenges for ensuring accurate translation across cultural contexts. △ Less

Submitted 14 May, 2024; originally announced May 2024.

Comments: Accepted at ACM CUI '24

Comparing Perceptions of Static and Adaptive Proactive Speech Agents

Authors: Justin Edwards, Philip R. Doyle, Holly P. Branigan, Benjamin R. Cowan

Abstract: A growing literature on speech interruptions describes how people interrupt one another with speech, but these behaviours have not yet been implemented in the design of artificial agents which interrupt. Perceptions of a prototype proactive speech agent which adapts its speech to both urgency and to the difficulty of the ongoing task it interrupts are compared against perceptions of a static proac… ▽ More A growing literature on speech interruptions describes how people interrupt one another with speech, but these behaviours have not yet been implemented in the design of artificial agents which interrupt. Perceptions of a prototype proactive speech agent which adapts its speech to both urgency and to the difficulty of the ongoing task it interrupts are compared against perceptions of a static proactive agent which does not. The study hypothesises that adaptive proactive speech modelled on human speech interruptions will lead to partner models which consider the proactive agent as a stronger conversational partner than a static agent, and that interruptions initiated by an adaptive agent will be judged as better timed and more appropriately asked. These hypotheses are all rejected however, as quantitative analysis reveals that participants view the adaptive agent as a poorer dialogue partner than the static agent and as less appropriate in the style it interrupts. Qualitative analysis sheds light on the source of this surprising finding, as participants see the adaptive agent as less socially appropriate and as less consistent in its interactions than the static agent. △ Less

Submitted 14 May, 2024; v1 submitted 13 May, 2024; originally announced May 2024.

Comments: Accepted to CUI 2024 - 6th Conference on Conversational User Interfaces

Listening to the Voices: Describing Ethical Caveats of Conversational User Interfaces According to Experts and Frequent Users

Authors: Thomas Mildner, Orla Cooney, Anna-Maria Meck, Marion Bartl, Gian-Luca Savino, Philip R. Doyle, Diego Garaialde, Leigh Clark, John Sloan, Nina Wenig, Rainer Malaka, Jasmin Niess

Abstract: Advances in natural language processing and understanding have led to a rapid growth in the popularity of conversational user interfaces (CUIs). While CUIs introduce novel benefits, they also yield risks that may exploit people's trust. Although research looking at unethical design deployed through graphical user interfaces (GUIs) established a thorough understanding of so-called dark patterns, th… ▽ More Advances in natural language processing and understanding have led to a rapid growth in the popularity of conversational user interfaces (CUIs). While CUIs introduce novel benefits, they also yield risks that may exploit people's trust. Although research looking at unethical design deployed through graphical user interfaces (GUIs) established a thorough understanding of so-called dark patterns, there is a need to continue this discourse within the CUI community to understand potentially problematic interactions. Addressing this gap, we interviewed 27 participants from three cohorts: researchers, practitioners, and frequent users of CUIs. Applying thematic analysis, we construct five themes reflecting each cohort's insights about ethical design challenges and introduce the CUI Expectation Cycle, bridging system capabilities and user expectations while considering each theme's ethical caveats. This research aims to inform future development of CUIs to consider ethical constraints while adopting a human-centred approach. △ Less

Submitted 26 January, 2024; originally announced January 2024.

Comments: 18 pages; 4 tables; and 1 figure. This is the author's version and pre-print of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record will be published in Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24), May 11--16, 2024, Honolulu, HI, USA, https://doi.org/https://doi.org/10.1145/3613904.3642542

ACM Class: H.1.2; K.4

The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners

Authors: Philip R. Doyle, Iona Gessinger, Justin Edwards, Leigh Clark, Odile Dumbleton, Diego Garaialde, Daniel Rough, Anna Bleakley, Holly P. Branigan, Benjamin R. Cowan

Abstract: Recent work has looked to understand user perceptions of speech agent capabilities as dialogue partners (termed partner models), and how this affects user interaction. Yet, currently partner model effects are inferred from language production as no metrics are available to quantify these subjective perceptions more directly. Through three studies, we develop and validate the Partner Modelling Ques… ▽ More Recent work has looked to understand user perceptions of speech agent capabilities as dialogue partners (termed partner models), and how this affects user interaction. Yet, currently partner model effects are inferred from language production as no metrics are available to quantify these subjective perceptions more directly. Through three studies, we develop and validate the Partner Modelling Questionnaire (PMQ): an 18-item self-report semantic differential scale designed to reliably measure people's partner models of non-embodied speech interfaces. Through principal component analysis and confirmatory factor analysis, we show that the PMQ scale consists of three factors: communicative competence and dependability, human-likeness in communication, and communicative flexibility. Our studies show that the measure consistently demonstrates good internal reliability, strong test-retest reliability over 12 and 4-week intervals, and predictable convergent/divergent validity. Based on our findings we discuss the multidimensional nature of partner models, whilst identifying key future research avenues that the development of the PMQ facilitates. Notably, this includes the need to identify the activation, sensitivity, and dynamism of partner models in speech interface interaction. △ Less

Submitted 14 August, 2023; originally announced August 2023.

Comments: Submitted (TOCHI)

Defending Against the Dark Arts: Recognising Dark Patterns in Social Media

Authors: Thomas Mildner, Merle Freye, Gian-Luca Savino, Philip R. Doyle, Benjamin R. Cowan, Rainer Malaka

Abstract: Interest in unethical user interfaces has grown in HCI over recent years, with researchers identifying malicious design strategies referred to as ''dark patterns''. While such strategies have been described in numerous domains, we lack a thorough understanding of how they operate in social networking services (SNSs). Pivoting towards regulations against such practices, we address this gap by offer… ▽ More Interest in unethical user interfaces has grown in HCI over recent years, with researchers identifying malicious design strategies referred to as ''dark patterns''. While such strategies have been described in numerous domains, we lack a thorough understanding of how they operate in social networking services (SNSs). Pivoting towards regulations against such practices, we address this gap by offering novel insights into the types of dark patterns deployed in SNSs and people's ability to recognise them across four widely used mobile SNS applications. Following a cognitive walkthrough, experts (N=6) could identify instances of dark patterns in all four SNSs, including co-occurrences. Based on the results, we designed a novel rating procedure for evaluating the malice of interfaces. Our evaluation shows that regular users (N=193) could differentiate between interfaces featuring dark patterns and those without. Such rating procedures could support policymakers' current moves to regulate deceptive and manipulative designs in online interfaces. △ Less

Submitted 22 May, 2023; originally announced May 2023.

Comments: 13 pages; 5 tables; and 11 figures. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Designing Interactive Systems Conference (DIS '23), July 10-14, 2023, Pittsburgh, PA, USA, https: //doi.org/10.1145/3563657.3595964

ACM Class: H.1.2; K.4.1

About Engaging and Governing Strategies: A Thematic Analysis of Dark Patterns in Social Networking Services

Authors: Thomas Mildner, Gian-Luca Savino, Philip R. Doyle, Benjamin R. Cowan, Rainer Malaka

Abstract: Research in HCI has shown a growing interest in unethical design practices across numerous domains, often referred to as ``dark patterns''. There is, however, a gap in related literature regarding social networking services (SNSs). In this context, studies emphasise a lack of users' self-determination regarding control over personal data and time spent on SNSs. We collected over 16 hours of screen… ▽ More Research in HCI has shown a growing interest in unethical design practices across numerous domains, often referred to as ``dark patterns''. There is, however, a gap in related literature regarding social networking services (SNSs). In this context, studies emphasise a lack of users' self-determination regarding control over personal data and time spent on SNSs. We collected over 16 hours of screen recordings from Facebook's, Instagram's, TikTok's, and Twitter's mobile applications to understand how dark patterns manifest in these SNSs. For this task, we turned towards HCI experts to mitigate possible difficulties of non-expert participants in recognising dark patterns, as prior studies have noticed. Supported by the recordings, two authors of this paper conducted a thematic analysis based on previously described taxonomies, manually classifying the recorded material while delivering two key findings: We observed which instances occur in SNSs and identified two strategies - engaging and governing - with five dark patterns undiscovered before. △ Less

Submitted 1 March, 2023; originally announced March 2023.

Comments: In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems; April 23--28, 2023; Hamburg, Germany; 16 pages, 9 figures

ACM Class: H.1.2; K.4.1

Journal ref: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23)

CUI @ Auto-UI: Exploring the Fortunate and Unfortunate Futures of Conversational Automotive User Interfaces

Authors: Justin Edwards, Philipp Wintersberger, Leigh Clark, Daniel Rough, Philip R Doyle, Victoria Banks, Adam Wyner, Christian P. Janssen, Benjamin R. Cowan

Abstract: This work aims to connect the Automotive User Interfaces (Auto-UI) and Conversational User Interfaces (CUI) communities through discussion of their shared view of the future of automotive conversational user interfaces. The workshop aims to encourage creative consideration of optimistic and pessimistic futures, encouraging attendees to explore the opportunities and barriers that lie ahead through… ▽ More This work aims to connect the Automotive User Interfaces (Auto-UI) and Conversational User Interfaces (CUI) communities through discussion of their shared view of the future of automotive conversational user interfaces. The workshop aims to encourage creative consideration of optimistic and pessimistic futures, encouraging attendees to explore the opportunities and barriers that lie ahead through a game. Considerations of the future will be mapped out in greater detail through the drafting of research agendas, by which attendees will get to know each other's expertise and networks of resources. The two day workshop, consisting of two 90-minute sessions, will facilitate greater communication and collaboration between these communities, connecting researchers to work together to influence the futures they imagine in the workshop. △ Less

Submitted 19 October, 2021; originally announced October 2021.

Comments: Workshop published and presented at Automotive User Interfaces 2021 (AutoUI 21)

Bridging Social Distance During Social Distancing: Exploring Social Talk and Remote Collegiality in Video Conferencing

Authors: Anna Bleakley, Daniel Rough, Justin Edwards, Philip R. Doyle, Odile Dumbleton, Leigh Clark, Sean Rintel, Vincent Wade, Benjamin R. Cowan

Abstract: Video conferencing systems have long facilitated work-related conversations among remote teams. However, social distancing due to the COVID-19 pandemic has forced colleagues to use video conferencing platforms to additionally fulfil social needs. Social talk, or informal talk, is an important workplace practice that is used to build and maintain bonds in everyday interactions among colleagues. Cur… ▽ More Video conferencing systems have long facilitated work-related conversations among remote teams. However, social distancing due to the COVID-19 pandemic has forced colleagues to use video conferencing platforms to additionally fulfil social needs. Social talk, or informal talk, is an important workplace practice that is used to build and maintain bonds in everyday interactions among colleagues. Currently, there is a limited understanding of how video conferencing facilitates multiparty social interactions among colleagues. In our paper, we examine social talk practices during the COVID-19 pandemic among remote colleagues through semi-structured interviews. We uncovered three key themes in our interviews, discussing 1) the changing purposes and opportunities afforded by using video conferencing for social talk with colleagues, 2) how the nature of existing relationships and status of colleagues influences social conversations and 3) the challenges and changing conversational norms around politeness and etiquette when using video conferencing to hold social conversations. We discuss these results in relation to the impact that video conferencing tools have on remote social talk between colleagues and outline design and best practice considerations for multiparty videoconferencing social talk in the workplace. △ Less

Submitted 30 September, 2021; originally announced September 2021.

Comments: Accepted to Journal of Human-computer Interaction Special Issue on the Future of Remote Work: Responses to the Pandemic

What Do We See in Them? Identifying Dimensions of Partner Models for Speech Interfaces Using a Psycholexical Approach

Authors: Philip R Doyle, Leigh Clark, Benjamin R Cowan

Abstract: Perceptions of system competence and communicative ability, termed partner models, play a significant role in speech interface interaction. Yet we do not know what the core dimensions of this concept are. Taking a psycholexical approach, our paper is the first to identify the key dimensions that define partner models in speech agent interaction. Through a repertory grid study (N=21), a review of k… ▽ More Perceptions of system competence and communicative ability, termed partner models, play a significant role in speech interface interaction. Yet we do not know what the core dimensions of this concept are. Taking a psycholexical approach, our paper is the first to identify the key dimensions that define partner models in speech agent interaction. Through a repertory grid study (N=21), a review of key subjective questionnaires, an expert review of resulting word pairs and an online study of 356 user of speech interfaces, we identify three key dimensions that make up a users' partner model: 1) perceptions toward competence and capability; 2) assessment of human-likeness; and 3) a system's perceived cognitive flexibility. We discuss the implications for partner modelling as a concept, emphasising the importance of salience and the dynamic nature of these perceptions. △ Less

Submitted 16 April, 2021; v1 submitted 3 February, 2021; originally announced February 2021.

Comments: Pre-print version. 14 pages (inc. bib), 3 figures, 5 tables

Mental Workload and Language Production in Non-Native Speaker IPA Interaction

Authors: Yunhan Wu, Justin Edwards, Orla Cooney, Anna Bleakley, Philip R. Doyle, Leigh Clark, Daniel Rough, Benjamin R. Cowan

Abstract: Through proliferation on smartphones and smart speakers, intelligent personal assistants (IPAs) have made speech a common interaction modality. Yet, due to linguistic coverage and varying levels of functionality, many speakers engage with IPAs using a non-native language. This may impact the mental workload and pattern of language production displayed by non-native speakers. We present a mixed-des… ▽ More Through proliferation on smartphones and smart speakers, intelligent personal assistants (IPAs) have made speech a common interaction modality. Yet, due to linguistic coverage and varying levels of functionality, many speakers engage with IPAs using a non-native language. This may impact the mental workload and pattern of language production displayed by non-native speakers. We present a mixed-design experiment, wherein native (L1) and non-native (L2) English speakers completed tasks with IPAs through smartphones and smart speakers. We found significantly higher mental workload for L2 speakers during IPA interactions. Contrary to our hypotheses, we found no significant differences between L1 and L2 speakers in terms of number of turns, lexical complexity, diversity, or lexical adaptation when encountering errors. These findings are discussed in relation to language production and processing load increases for L2 speakers in IPA interaction. △ Less

Submitted 11 June, 2020; originally announced June 2020.

Comments: Accepted at CUI 2020

See what I'm saying? Comparing Intelligent Personal Assistant use for Native and Non-Native Language Speakers

Authors: Yunhan Wu, Daniel Rough, Anna Bleakley, Justin Edwards, Orla Cooney, Philip R. Doyle, Leigh Clark, Benjamin R. Cowan

Abstract: Limited linguistic coverage for Intelligent Personal Assistants (IPAs) means that many interact in a non-native language. Yet we know little about how IPAs currently support or hinder these users. Through native (L1) and non-native (L2) English speakers interacting with Google Assistant on a smartphone and smart speaker, we aim to understand this more deeply. Interviews revealed that L2 speakers p… ▽ More Limited linguistic coverage for Intelligent Personal Assistants (IPAs) means that many interact in a non-native language. Yet we know little about how IPAs currently support or hinder these users. Through native (L1) and non-native (L2) English speakers interacting with Google Assistant on a smartphone and smart speaker, we aim to understand this more deeply. Interviews revealed that L2 speakers prioritised utterance planning around perceived linguistic limitations, as opposed to L1 speakers prioritising succinctness because of system limitations. L2 speakers see IPAs as insensitive to linguistic needs resulting in failed interaction. L2 speakers clearly preferred using smartphones, as visual feedback supported diagnoses of communication breakdowns whilst allowing time to process query results. Conversely, L1 speakers preferred smart speakers, with audio feedback being seen as sufficient. We discuss the need to tailor the IPA experience for L2 users, emphasising visual feedback whilst reducing the burden of language production. △ Less

Submitted 11 June, 2020; originally announced June 2020.

Comments: Accepted to Mobile HCI 2020

Transparency in Language Generation: Levels of Automation

Authors: Justin Edwards, Allison Perrone, Philip R. Doyle

Abstract: Language models and conversational systems are growing increasingly advanced, creating outputs that may be mistaken for humans. Consumers may thus be misled by advertising, media reports, or vagueness regarding the role of automation in the production of language. We propose a taxonomy of language automation, based on the SAE levels of driving automation, to establish a shared set of terms for des… ▽ More Language models and conversational systems are growing increasingly advanced, creating outputs that may be mistaken for humans. Consumers may thus be misled by advertising, media reports, or vagueness regarding the role of automation in the production of language. We propose a taxonomy of language automation, based on the SAE levels of driving automation, to establish a shared set of terms for describing automated language. It is our hope that the proposed taxonomy can increase transparency in this rapidly advancing field. △ Less

Submitted 11 June, 2020; originally announced June 2020.

Comments: Accepted for publication at CUI 2020

Mapping Perceptions of Humanness in Speech-Based Intelligent Personal Assistant Interaction

Authors: Philip R. Doyle, Justin Edwards, Odile Dumbleton, Leigh Clark, Benjamin R. Cowan

Abstract: Humanness is core to speech interface design. Yet little is known about how users conceptualise perceptions of humanness and how people define their interaction with speech interfaces through this. To map these perceptions n=21 participants held dialogues with a human and two speech interface based intelligent personal assistants, and then reflected and compared their experiences using the reperto… ▽ More Humanness is core to speech interface design. Yet little is known about how users conceptualise perceptions of humanness and how people define their interaction with speech interfaces through this. To map these perceptions n=21 participants held dialogues with a human and two speech interface based intelligent personal assistants, and then reflected and compared their experiences using the repertory grid technique. Analysis of the constructs show that perceptions of humanness are multidimensional, focusing on eight key themes: partner knowledge set, interpersonal connection, linguistic content, partner performance and capabilities, conversational interaction, partner identity and role, vocal qualities and behavioral affordances. Through these themes, it is clear that users define the capabilities of speech interfaces differently to humans, seeing them as more formal, fact based, impersonal and less authentic. Based on the findings, we discuss how the themes help to scaffold, categorise and target research and design efforts, considering the appropriateness of emulating humanness. △ Less

Submitted 29 July, 2019; v1 submitted 26 July, 2019; originally announced July 2019.