-
Explicit and Implicit Large Language Model Personas Generate Opinions but Fail to Replicate Deeper Perceptions and Biases
Authors:
Salvatore Giorgi,
Tingting Liu,
Ankit Aich,
Kelsey Isman,
Garrick Sherman,
Zachary Fried,
João Sedoc,
Lyle H. Ungar,
Brenda Curtis
Abstract:
Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. However, these tasks are highly subjective and dependent on human factors, such as one's environment, attitudes, beliefs, and lived experiences. Thus, employing LLMs (which do not have such human factors) in these tasks may re…
▽ More
Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. However, these tasks are highly subjective and dependent on human factors, such as one's environment, attitudes, beliefs, and lived experiences. Thus, employing LLMs (which do not have such human factors) in these tasks may result in a lack of variation in data, failing to reflect the diversity of human experiences. In this paper, we examine the role of prompting LLMs with human-like personas and asking the models to answer as if they were a specific human. This is done explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. The LLM personas are then evaluated via (1) subjective annotation task (e.g., detecting toxicity) and (2) a belief generation task, where both tasks are known to vary across human factors. We examine the impact of explicit vs. implicit personas and investigate which human factors LLMs recognize and respond to. Results show that LLM personas show mixed results when reproducing known human biases, but generate generally fail to demonstrate implicit biases. We conclude that LLMs lack the intrinsic cognitive mechanisms of human thought, while capturing the statistical patterns of how people speak, which may restrict their effectiveness in complex social science applications.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Graph Koopman Autoencoder for Predictive Covert Communication Against UAV Surveillance
Authors:
Sivaram Krishnan,
Jihong Park,
Gregory Sherman,
Benjamin Campbell,
Jinho Choi
Abstract:
Low Probability of Detection (LPD) communication aims to obscure the very presence of radio frequency (RF) signals, going beyond just hiding the content of the communication. However, the use of Unmanned Aerial Vehicles (UAVs) introduces a challenge, as UAVs can detect RF signals from the ground by hovering over specific areas of interest. With the growing utilization of UAVs in modern surveillanc…
▽ More
Low Probability of Detection (LPD) communication aims to obscure the very presence of radio frequency (RF) signals, going beyond just hiding the content of the communication. However, the use of Unmanned Aerial Vehicles (UAVs) introduces a challenge, as UAVs can detect RF signals from the ground by hovering over specific areas of interest. With the growing utilization of UAVs in modern surveillance, there is a crucial need for a thorough understanding of their unknown nonlinear dynamic trajectories to effectively implement LPD communication. Unfortunately, this critical information is often not readily available, posing a significant hurdle in LPD communication. To address this issue, we consider a case-study for enabling terrestrial LPD communication in the presence of multiple UAVs that are engaged in surveillance. We introduce a novel framework that combines graph neural networks (GNN) with Koopman theory to predict the trajectories of multiple fixed-wing UAVs over an extended prediction horizon. Using the predicted UAV locations, we enable LPD communication in a terrestrial ad-hoc network by controlling nodes' transmit powers to keep the received power at UAVs' predicted locations minimized. Our extensive simulations validate the efficacy of the proposed framework in accurately predicting the trajectories of multiple UAVs, thereby effectively establishing LPD communication.
△ Less
Submitted 23 January, 2024;
originally announced February 2024.
-
Language-based Valence and Arousal Expressions between the United States and China: a Cross-Cultural Examination
Authors:
Young-Min Cho,
Dandan Pang,
Stuti Thapa,
Garrick Sherman,
Lyle Ungar,
Louis Tay,
Sharath Chandra Guntuku
Abstract:
Although affective expressions of individuals have been extensively studied using social media, research has primarily focused on the Western context. There are substantial differences among cultures that contribute to their affective expressions. This paper examines the differences between Twitter (X) in the United States and Sina Weibo posts in China on two primary dimensions of affect - valence…
▽ More
Although affective expressions of individuals have been extensively studied using social media, research has primarily focused on the Western context. There are substantial differences among cultures that contribute to their affective expressions. This paper examines the differences between Twitter (X) in the United States and Sina Weibo posts in China on two primary dimensions of affect - valence and arousal. We study the difference in the functional relationship between arousal and valence (so-called V-shaped) among individuals in the US and China and explore the associated content differences. Furthermore, we correlate word usage and topics in both platforms to interpret their differences. We observe that for Twitter users, the variation in emotional intensity is less distinct between negative and positive emotions compared to Weibo users, and there is a sharper escalation in arousal corresponding with heightened emotions. From language features, we discover that affective expressions are associated with personal life and feelings on Twitter, while on Weibo such discussions are about socio-political topics in the society. These results suggest a West-East difference in the V-shaped relationship between valence and arousal of affective expressions on social media influenced by content differences. Our findings have implications for applications and theories related to cultural differences in affective expressions.
△ Less
Submitted 30 July, 2024; v1 submitted 10 January, 2024;
originally announced January 2024.
-
Historical patterns of rice farming explain modern-day language use in China and Japan more than modernization and urbanization
Authors:
Sharath Chandra Guntuku,
Thomas Talhelm,
Garrick Sherman,
Angel Fan,
Salvatore Giorgi,
Liuqing Wei,
Lyle H. Ungar
Abstract:
We used natural language processing to analyze a billion words to study cultural differences on Weibo, one of China's largest social media platforms. We compared predictions from two common explanations about cultural differences in China (economic development and urban-rural differences) against the less-obvious legacy of rice versus wheat farming. Rice farmers had to coordinate shared irrigation…
▽ More
We used natural language processing to analyze a billion words to study cultural differences on Weibo, one of China's largest social media platforms. We compared predictions from two common explanations about cultural differences in China (economic development and urban-rural differences) against the less-obvious legacy of rice versus wheat farming. Rice farmers had to coordinate shared irrigation networks and exchange labor to cope with higher labor requirements. In contrast, wheat relied on rainfall and required half as much labor. We test whether this legacy made southern China more interdependent. Across all word categories, rice explained twice as much variance as economic development and urbanization. Rice areas used more words reflecting tight social ties, holistic thought, and a cautious, prevention orientation. We then used Twitter data comparing prefectures in Japan, which largely replicated the results from China. This provides crucial evidence of the rice theory in a different nation, language, and platform.
△ Less
Submitted 29 August, 2023;
originally announced August 2023.
-
Federated Graph Learning for Low Probability of Detection in Wireless Ad-Hoc Networks
Authors:
Sivaram Krishnan,
Jihong Park,
Subhash Sagar,
Gregory Sherman,
Benjamin Campbell,
Jinho Choi
Abstract:
Low probability of detection (LPD) has recently emerged as a means to enhance the privacy and security of wireless networks. Unlike existing wireless security techniques, LPD measures aim to conceal the entire existence of wireless communication instead of safeguarding the information transmitted from users. Motivated by LPD communication, in this paper, we study a privacy-preserving and distribut…
▽ More
Low probability of detection (LPD) has recently emerged as a means to enhance the privacy and security of wireless networks. Unlike existing wireless security techniques, LPD measures aim to conceal the entire existence of wireless communication instead of safeguarding the information transmitted from users. Motivated by LPD communication, in this paper, we study a privacy-preserving and distributed framework based on graph neural networks to minimise the detectability of a wireless ad-hoc network as a whole and predict an optimal communication region for each node in the wireless network, allowing them to communicate while remaining undetected from external actors. We also demonstrate the effectiveness of the proposed method in terms of two performance measures, i.e., mean absolute error and median absolute error.
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
Lived Experience Matters: Automatic Detection of Stigma on Social Media Toward People Who Use Substances
Authors:
Salvatore Giorgi,
Douglas Bellew,
Daniel Roy Sadek Habib,
Garrick Sherman,
Joao Sedoc,
Chase Smitterberg,
Amanda Devoto,
McKenzie Himelein-Wachowiak,
Brenda Curtis
Abstract:
Stigma toward people who use substances (PWUS) is a leading barrier to seeking treatment.Further, those in treatment are more likely to drop out if they experience higher levels of stigmatization. While related concepts of hate speech and toxicity, including those targeted toward vulnerable populations, have been the focus of automatic content moderation research, stigma and, in particular, people…
▽ More
Stigma toward people who use substances (PWUS) is a leading barrier to seeking treatment.Further, those in treatment are more likely to drop out if they experience higher levels of stigmatization. While related concepts of hate speech and toxicity, including those targeted toward vulnerable populations, have been the focus of automatic content moderation research, stigma and, in particular, people who use substances have not. This paper explores stigma toward PWUS using a data set of roughly 5,000 public Reddit posts. We performed a crowd-sourced annotation task where workers are asked to annotate each post for the presence of stigma toward PWUS and answer a series of questions related to their experiences with substance use. Results show that workers who use substances or know someone with a substance use disorder are more likely to rate a post as stigmatizing. Building on this, we use a supervised machine learning framework that centers workers with lived substance use experience to label each Reddit post as stigmatizing. Modeling person-level demographics in addition to comment-level language results in a classification accuracy (as measured by AUC) of 0.69 -- a 17% increase over modeling language alone. Finally, we explore the linguist cues which distinguish stigmatizing content: PWUS substances and those who don't agree that language around othering ("people", "they") and terms like "addict" are stigmatizing, while PWUS (as opposed to those who do not) find discussions around specific substances more stigmatizing. Our findings offer insights into the nature of perceived stigma in substance use. Additionally, these results further establish the subjective nature of such machine learning tasks, highlighting the need for understanding their social contexts.
△ Less
Submitted 16 July, 2023; v1 submitted 3 February, 2023;
originally announced February 2023.
-
Social Media Reveals Urban-Rural Differences in Stress across China
Authors:
Jesse Cui,
Tingdan Zhang,
Kokil Jaidka,
Dandan Pang,
Garrick Sherman,
Vinit Jakhetiya,
Lyle Ungar,
Sharath Chandra Guntuku
Abstract:
Modeling differential stress expressions in urban and rural regions in China can provide a better understanding of the effects of urbanization on psychological well-being in a country that has rapidly grown economically in the last two decades. This paper studies linguistic differences in the experiences and expressions of stress in urban-rural China from Weibo posts from over 65,000 users across…
▽ More
Modeling differential stress expressions in urban and rural regions in China can provide a better understanding of the effects of urbanization on psychological well-being in a country that has rapidly grown economically in the last two decades. This paper studies linguistic differences in the experiences and expressions of stress in urban-rural China from Weibo posts from over 65,000 users across 329 counties using hierarchical mixed-effects models. We analyzed phrases, topical themes, and psycho-linguistic word choices in Weibo posts mentioning stress to better understand appraisal differences surrounding psychological stress in urban and rural communities in China; we then compared them with large-scale polls from Gallup. After controlling for socioeconomic and gender differences, we found that rural communities tend to express stress in emotional and personal themes such as relationships, health, and opportunity while users in urban areas express stress using relative, temporal, and external themes such as work, politics, and economics. These differences exist beyond controlling for GDP and urbanization, indicating a fundamentally different lifestyle between rural and urban residents in very specific environments, arguably having different sources of stress. We found corroborative trends in physical, financial, and social wellness with urbanization in Gallup polls.
△ Less
Submitted 3 November, 2021; v1 submitted 19 October, 2021;
originally announced October 2021.
-
Comparing Decison Support Tools for Cargo Screening Processes
Authors:
Peer-Olaf Siebers,
Galina Sherman,
Uwe Aickelin,
David Menachof
Abstract:
When planning to change operations at ports there are two key stake holders with very different interests involved in the decision making processes. Port operators are attentive to their standards, a smooth service flow and economic viability while border agencies are concerned about national security. The time taken for security checks often interferes with the compliance to service standards tha…
▽ More
When planning to change operations at ports there are two key stake holders with very different interests involved in the decision making processes. Port operators are attentive to their standards, a smooth service flow and economic viability while border agencies are concerned about national security. The time taken for security checks often interferes with the compliance to service standards that port operators would like to achieve. Decision support tools as for example Cost-Benefit Analysis or Multi Criteria Analysis are useful helpers to better understand the impact of changes to a system. They allow investigating future scenarios and helping to find solutions that are acceptable for all parties involved in port operations. In this paper we evaluate two different modelling methods, namely scenario analysis and discrete event simulation. These are useful for driving the decision support tools (i.e. they provide the inputs the decision support tools require). Our aims are, on the one hand, to guide the reader through the modelling processes and, on the other hand, to demonstrate what kind of decision support information one can obtain from the different modelling methods presented.
△ Less
Submitted 2 July, 2013;
originally announced July 2013.
-
Towards modelling cost and risks of infrequent events in the cargo screening process
Authors:
Galina Sherman,
David Menachof,
Peer-Olaf Siebers,
Uwe Aickelin
Abstract:
We introduce a simulation model of the port of Calais with a focus on the operation of immigration controls. Our aim is to compare the cost and benefits of different screening policies. Methodologically, we are trying to understand the limits of discrete event simulation of rare events. When will they become 'too rare' for simulation to give meaningful results?
We introduce a simulation model of the port of Calais with a focus on the operation of immigration controls. Our aim is to compare the cost and benefits of different screening policies. Methodologically, we are trying to understand the limits of discrete event simulation of rare events. When will they become 'too rare' for simulation to give meaningful results?
△ Less
Submitted 21 June, 2013;
originally announced June 2013.
-
Scenario Analysis, Decision Trees and Simulation for Cost Benefit Analysis of the Cargo Screening Process
Authors:
Galina Sherman,
Peer-Olaf Siebers,
Uwe Aickelin,
David Menachof
Abstract:
In this paper we present our ideas for conducting a cost benefit analysis by using three different methods: scenario analysis, decision trees and simulation. Then we introduce our case study and examine these methods in a real world situation. We show how these tools can be used and what the results are for each of them. Our aim is to conduct a comparison of these different probabilistic methods o…
▽ More
In this paper we present our ideas for conducting a cost benefit analysis by using three different methods: scenario analysis, decision trees and simulation. Then we introduce our case study and examine these methods in a real world situation. We show how these tools can be used and what the results are for each of them. Our aim is to conduct a comparison of these different probabilistic methods of estimating costs for port security risk assessment studies. Methodologically, we are trying to understand the limits of all the tools mentioned above by focusing on rare events.
△ Less
Submitted 21 June, 2013;
originally announced June 2013.
-
Validation of a Microsimulation of the Port of Dover
Authors:
Chris Roadknight,
Uwe Aickelin,
Galina Sherman
Abstract:
Modelling and simulating the traffic of heavily used but secure environments such as seaports and airports is of increasing importance. Errors made when simulating these environments can have long standing economic, social and environmental implications. This paper discusses issues and problems that may arise when designing a simulation strategy. Data for the Port is presented, methods for lightwe…
▽ More
Modelling and simulating the traffic of heavily used but secure environments such as seaports and airports is of increasing importance. Errors made when simulating these environments can have long standing economic, social and environmental implications. This paper discusses issues and problems that may arise when designing a simulation strategy. Data for the Port is presented, methods for lightweight vehicle assessment that can be used to calibrate and validate simulations are also discussed along with a diagnosis of overcalibration issues. We show that decisions about where the intelligence lies in a system has important repercussions for the reliability of system statistics. Finally, conclusions are drawn about how microsimulations can be moved forward as a robust planning tool for the 21st century.
△ Less
Submitted 31 May, 2013;
originally announced May 2013.
-
Evaluating Different Cost-Benefit Analysis Methods for Port Security Operations
Authors:
Galina Sherman,
Peer-Olaf Siebers,
David Menachof,
Uwe Aickelin
Abstract:
Service industries, such as ports, are attentive to their standards, a smooth service flow and economic viability. Cost benefit analysis has proven itself as a useful tool to support this type of decision making; it has been used by businesses and governmental agencies for many years. In this book chapter we demonstrate different modelling methods that are used for estimating input factors require…
▽ More
Service industries, such as ports, are attentive to their standards, a smooth service flow and economic viability. Cost benefit analysis has proven itself as a useful tool to support this type of decision making; it has been used by businesses and governmental agencies for many years. In this book chapter we demonstrate different modelling methods that are used for estimating input factors required for conducting cost benefit analysis based on a single case study. These methods are: scenario analysis, decision trees, Monte-Carlo simulation modelling and discrete event simulation modelling. Our aims are, on the one hand, to guide the analyst through the modelling processes and, on the other hand, to demonstrate what additional decision support information can be obtained from applying each of these modelling methods.
△ Less
Submitted 31 May, 2013;
originally announced May 2013.
-
Modelling and Analysing Cargo Screening Processes: A Project Outline
Authors:
Peer-Olaf Siebers,
Uwe Aickelin,
David Menachof,
Galina Sherman,
Peter Zimmerman
Abstract:
The efficiency of current cargo screening processes at sea and air ports is unknown as no benchmarks exists against which they could be measured. Some manufacturer benchmarks exist for individual sensors but we have not found any benchmarks that take a holistic view of the screening procedures assessing a combination of sensors and also taking operator variability into account. Just adding up reso…
▽ More
The efficiency of current cargo screening processes at sea and air ports is unknown as no benchmarks exists against which they could be measured. Some manufacturer benchmarks exist for individual sensors but we have not found any benchmarks that take a holistic view of the screening procedures assessing a combination of sensors and also taking operator variability into account. Just adding up resources and manpower used is not an effective way for assessing systems where human decision-making and operator compliance to rules play a vital role. For such systems more advanced assessment methods need to be used, taking into account that the cargo screening process is of a dynamic and stochastic nature. Our project aim is to develop a decision support tool (cargo-screening system simulator) that will map the right technology and manpower to the right commodity-threat combination in order to maximize detection rates. In this paper we present a project outline and highlight the research challenges we have identified so far. In addition we introduce our first case study, where we investigate the cargo screening process at the ferry port in Calais.
△ Less
Submitted 30 May, 2013;
originally announced May 2013.
-
Development of a Cargo Screening Process Simulator: A First Approach
Authors:
Peer-Olaf Siebers,
Galina Sherman,
Uwe Aickelin
Abstract:
The efficiency of current cargo screening processes at sea and air ports is largely unknown as few benchmarks exists against which they could be measured. Some manufacturers provide benchmarks for individual sensors but we found no benchmarks that take a holistic view of the overall screening procedures and no benchmarks that take operator variability into account. Just adding up resources and man…
▽ More
The efficiency of current cargo screening processes at sea and air ports is largely unknown as few benchmarks exists against which they could be measured. Some manufacturers provide benchmarks for individual sensors but we found no benchmarks that take a holistic view of the overall screening procedures and no benchmarks that take operator variability into account. Just adding up resources and manpower used is not an effective way for assessing systems where human decision-making and operator compliance to rules play a vital role. Our aim is to develop a decision support tool (cargo-screening system simulator) that will map the right technology and manpower to the right commodity-threat combination in order to maximise detection rates. In this paper we present our ideas for developing such a system and highlight the research challenges we have identified. Then we introduce our first case study and report on the progress we have made so far.
△ Less
Submitted 22 March, 2010;
originally announced March 2010.
