How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO
Authors:
Man Tik Ng,
Hui Tung Tse,
Jen-tse Huang,
Jingjing Li,
Wenxuan Wang,
Michael R. Lyu
Abstract:
The role-play ability of Large Language Models (LLMs) has emerged as a popular research direction. However, existing studies focus on imitating well-known public figures or fictional characters, overlooking the potential for simulating ordinary individuals. Such an oversight limits the potential for advancements in digital human clones and non-player characters in video games. To bridge this gap,…
▽ More
The role-play ability of Large Language Models (LLMs) has emerged as a popular research direction. However, existing studies focus on imitating well-known public figures or fictional characters, overlooking the potential for simulating ordinary individuals. Such an oversight limits the potential for advancements in digital human clones and non-player characters in video games. To bridge this gap, we introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. Notably, our framework focuses on emulating average individuals rather than historical or fictional figures, presenting a unique advantage to apply the Turing Test. We evaluated three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models, alongside the online application GPTs from OpenAI. Our results demonstrate that GPT-4 more effectively deceives human evaluators, and GPTs achieves a leading success rate of 48.3%. Furthermore, we investigated whether LLMs could discern between human-generated and machine-generated texts. While GPT-4 can identify differences, it could not determine which texts were human-produced. Our code and results of reproducing the role-playing LLMs are made publicly available via https://github.com/CUHK-ARISE/ECHO.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
A Probabilistic Chemical Programmable Computer
Authors:
Abhishek Sharma,
Marcus Tze-Kiat Ng,
Juan Manuel Parrilla Gutierrez,
Yibin Jiang,
Leroy Cronin
Abstract:
The exponential growth of the power of modern digital computers is based upon the miniaturisation of vast nanoscale arrays of electronic switches, but this will be eventually constrained by fabrication limits and power dissipation. Chemical processes have the potential to scale beyond these limits performing computations through chemical reactions, yet the lack of well-defined programmability limi…
▽ More
The exponential growth of the power of modern digital computers is based upon the miniaturisation of vast nanoscale arrays of electronic switches, but this will be eventually constrained by fabrication limits and power dissipation. Chemical processes have the potential to scale beyond these limits performing computations through chemical reactions, yet the lack of well-defined programmability limits their scalability and performance. We present a hybrid digitally programmable chemical array as a probabilistic computational machine that uses chemical oscillators partitioned in interconnected cells as a computational substrate. This hybrid architecture performs efficient computation by distributing between chemical and digital domains together with error correction. The efficiency is gained by combining digital with probabilistic chemical logic based on nearest neighbour interactions and hysteresis effects. We demonstrated the implementation of one- and two- dimensional Chemical Cellular Automata and solutions to combinatorial optimization problems.
△ Less
Submitted 28 April, 2022;
originally announced April 2022.
