-
Soft Preference Optimization: Aligning Language Models to Expert Distributions
Authors:
Arsalan Sharifnassab,
Sina Ghiassian,
Saber Salehkaleybar,
Surya Kanoria,
Dale Schuurmans
Abstract:
We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than l…
▽ More
We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.
△ Less
Submitted 27 May, 2024; v1 submitted 30 April, 2024;
originally announced May 2024.
-
Automatic Music Playlist Generation via Simulation-based Reinforcement Learning
Authors:
Federico Tomasi,
Joseph Cauteruccio,
Surya Kanoria,
Kamil Ciosek,
Matteo Rinaldi,
Zhenwen Dai
Abstract:
Personalization of playlists is a common feature in music streaming services, but conventional techniques, such as collaborative filtering, rely on explicit assumptions regarding content quality to learn how to make recommendations. Such assumptions often result in misalignment between offline model objectives and online user satisfaction metrics. In this paper, we present a reinforcement learning…
▽ More
Personalization of playlists is a common feature in music streaming services, but conventional techniques, such as collaborative filtering, rely on explicit assumptions regarding content quality to learn how to make recommendations. Such assumptions often result in misalignment between offline model objectives and online user satisfaction metrics. In this paper, we present a reinforcement learning framework that solves for such limitations by directly optimizing for user satisfaction metrics via the use of a simulated playlist-generation environment. Using this simulator we develop and train a modified Deep Q-Network, the action head DQN (AH-DQN), in a manner that addresses the challenges imposed by the large state and action space of our RL formulation. The resulting policy is capable of making recommendations from large and dynamic sets of candidate items with the expectation of maximizing consumption metrics. We analyze and evaluate agents offline via simulations that use environment models trained on both public and proprietary streaming datasets. We show how these agents lead to better user-satisfaction metrics compared to baseline methods during online A/B tests. Finally, we demonstrate that performance assessments produced from our simulator are strongly correlated with observed online metric results.
△ Less
Submitted 13 October, 2023;
originally announced October 2023.
-
What to Learn, and How: Toward Effective Learning from Rationales
Authors:
Samuel Carton,
Surya Kanoria,
Chenhao Tan
Abstract:
Learning from rationales seeks to augment model prediction accuracy using human-annotated rationales (i.e. subsets of input tokens) that justify their chosen labels, often in the form of intermediate or multitask supervision. While intuitive, this idea has proven elusive in practice. We make two observations about human rationales via empirical analyses: 1) maximizing rationale supervision accurac…
▽ More
Learning from rationales seeks to augment model prediction accuracy using human-annotated rationales (i.e. subsets of input tokens) that justify their chosen labels, often in the form of intermediate or multitask supervision. While intuitive, this idea has proven elusive in practice. We make two observations about human rationales via empirical analyses: 1) maximizing rationale supervision accuracy is not necessarily the optimal objective for improving model accuracy; 2) human rationales vary in whether they provide sufficient information for the model to exploit for prediction. Building on these insights, we propose several novel loss functions and learning strategies, and evaluate their effectiveness on three datasets with human rationales. Our results demonstrate consistent improvements over baselines in both label and rationale accuracy, including a 3% accuracy improvement on MultiRC. Our work highlights the importance of understanding properties of human explanations and exploiting them accordingly in model training.
△ Less
Submitted 28 March, 2022; v1 submitted 30 November, 2021;
originally announced December 2021.
-
Paving the Way for Culturally Competent Robots: a Position Paper
Authors:
Barbara Bruno,
Nak Young Chong,
Hiroko Kamide,
Sanjeev Kanoria,
Jaeryoung Lee,
Yuto Lim,
Amit Kumar Pandey,
Chris Papadopoulos,
Irena Papadopoulos,
Federico Pecora,
Alessandro Saffiotti,
Antonio Sgorbissa
Abstract:
Cultural competence is a well known requirement for an effective healthcare, widely investigated in the nursing literature. We claim that personal assistive robots should likewise be culturally competent, aware of general cultural characteristics and of the different forms they take in different individuals, and sensitive to cultural differences while perceiving, reasoning, and acting. Drawing ins…
▽ More
Cultural competence is a well known requirement for an effective healthcare, widely investigated in the nursing literature. We claim that personal assistive robots should likewise be culturally competent, aware of general cultural characteristics and of the different forms they take in different individuals, and sensitive to cultural differences while perceiving, reasoning, and acting. Drawing inspiration from existing guidelines for culturally competent healthcare and the state-of-the-art in culturally competent robotics, we identify the key robot capabilities which enable culturally competent behaviours and discuss methodologies for their development and evaluation.
△ Less
Submitted 22 March, 2018;
originally announced March 2018.
-
The CARESSES EU-Japan project: making assistive robots culturally competent
Authors:
Barbara Bruno,
Nak Young Chong,
Hiroko Kamide,
Sanjeev Kanoria,
Jaeryoung Lee,
Yuto Lim,
Amit Kumar Pandey,
Chris Papadopoulos,
Irena Papadopoulos,
Federico Pecora,
Alessandro Saffiotti,
Antonio Sgorbissa
Abstract:
The nursing literature shows that cultural competence is an important requirement for effective healthcare. We claim that personal assistive robots should likewise be culturally competent, that is, they should be aware of general cultural characteristics and of the different forms they take in different individuals, and take these into account while perceiving, reasoning, and acting. The CARESSES…
▽ More
The nursing literature shows that cultural competence is an important requirement for effective healthcare. We claim that personal assistive robots should likewise be culturally competent, that is, they should be aware of general cultural characteristics and of the different forms they take in different individuals, and take these into account while perceiving, reasoning, and acting. The CARESSES project is an Europe-Japan collaborative effort that aims at designing, developing and evaluating culturally competent assistive robots. These robots will be able to adapt the way they behave, speak and interact to the cultural identity of the person they assist. This paper describes the approach taken in the CARESSES project, its initial steps, and its future plans.
△ Less
Submitted 21 August, 2017;
originally announced August 2017.
