Balancing the Scales: Reinforcement Learning for Fair Classification
Abstract
Fairness in classification tasks has traditionally focused on bias removal from neural representations, but recent trends favor algorithmic methods that embed fairness into the training process. These methods steer models towards fair performance, preventing potential elimination of valuable information that arises from representation manipulation. Reinforcement Learning (RL), with its capacity for learning through interaction and adjusting reward functions to encourage desired behaviors, emerges as a promising tool in this domain. In this paper, we explore the usage of RL to address bias in imbalanced classification by scaling the reward function to mitigate bias. We employ the contextual multi-armed bandit framework and adapt three popular RL algorithms to suit our objectives, demonstrating a novel approach to mitigating bias. 111Our code is available at https://github.com/watermeleon/RL_for_imbalanced_classification
1 Introduction
In recent years, the issue of bias and fairness in Artificial Intelligence and Natural Language Processing has received significant attention Mehrabi et al. (2021). In decision-making models such as classification algorithms, bias often stems directly from the training data leading to unfair outcomes between protected groups such as gender or race. To address this problem, previous work on fairness has focused on achieving representational fairness, so that the information of the protected groups is lost Ravfogel et al. (2020); Haghighatkhah et al. (2022). However, recent work has demonstrated no meaningful correlation between representational fairness and empirical fairness, i.e. fairness on downstream tasks Shen et al. (2022). To address empirical fairness directly, other work has explored the intersection of bias mitigation and class-imbalanced learning Subramanian et al. (2021). Class-imbalanced learning approaches aim to achieve fair performance by balancing the training data via sampling or reweighing the loss function.
At the same time, Reinforcement Learning (RL) has emerged as a promising alternative to traditional supervised learning methods for various NLP tasks, including syntactic parsing, conversational systems, and machine translation Uc-Cetina et al. (2023). Unlike traditional supervised learning methods, RL is not bound to binary labels and is trained directly on the continuous value of each input, as illustrated in Figure 1. RL agents can learn from sparse reward signals, receiving the rewards for the action they choose, not necessarily the correct one. By exploring the environment and adapting their behavior based on the received rewards, RL agents find optimal actions under varying state values. In the context of classification, RL has been adapted to mitigate class imbalance through a scaling component of the reward function for binary classification Lin et al. (2020). However, implementations considering more complex imbalances have remained largely unexplored.
In this work, we leverage RL to address fairness among protected groups in multi-class classification. First, we propose to frame the fair classification task as a Contextual Multi-Armed Bandit (CMAB) problem, see Figure 1 for an overview of our setup. To mitigate bias, we scale the reward function to counteract imbalances among protected groups within each class. We employ three different types of RL methods, each reflecting a key type of RL approach, and adapt them for our task. Additionally, we integrate the different scaling approaches into a supervised learning baseline to evaluate the effectiveness of our RL-based methods.
Experiments on two fair classification datasets demonstrate that our RL algorithms achieve competitive performance compared to existing baselines and that reward scaling is a powerful tool to mitigate bias in classification. We further investigate how stable reward scaling is under various class and subclass imbalances as well as various degrees of representational fairness. Notably, the deep RL algorithms perform best on the multi-class dataset, while the classical CMAB algorithm excels on the binary dataset. Moreover, our scaled supervised implementation exceeds existing implementations and shows state-of-the-art performance for multi-class classification.
2 Related work
2.1 Bias Mitigation
Research on mitigating bias can be divided in those that tackle the training data Wang et al. (2019), those that attempt to remove bias from representations Ravfogel et al. (2020); Haghighatkhah et al. (2022), and those that adjust the learning process Elazar and Goldberg (2018); Han et al. (2021). Within approaches that adjust the learning processes, we distinguish two main categories: those that add adversarial learners to ignore protected attributes Wadsworth et al. (2018), and more closely to our work, approaches that adjust the loss function to emphasize performance on minority classes.
Prior work that modified the training setup to increase fairness used methods such as down/upsampling Wang et al. (2019) and reweighting the loss function Höfler et al. (2005); Lahoti et al. (2020). Han et al. (2022a) evaluate both down-sampling and loss reweighting on two datasets for fair text classification. Both techniques are applied to align training with different definitions of fairness. Downsampling using the Equal Opportunity fairness metric demonstrated impressive results. In this paper, we take the first step to explore whether reward scaling in reinforcement learning can improve fairness in classification.
![Refer to caption](x1.png)
2.2 Reinforcement Learning for Classification
Literature on RL applications for classification predominantly considers the following two theoretical frameworks: Markov Decision Process (MDP) and the Contextual Multi-Armed Bandit (CMAB).
Early work by Wiering et al. (2011) casts classification as a sequential decision-making task, by introducing a classification variant of the MDP. In their setup agents manipulate memory cells to encode information by applying an action sequence on a single sample. They demonstrated competitive performance, but, remained limited to small tasks due to the computational complexity. Lin et al. (2020) extended this work, by introducing a variant of the classification MDP and applying a Deep Q-learning Network (DQN) to binary classification of images and texts. They focused on mitigating bias arising from class imbalance by scaling the rewards inversely proportional to the class frequency. However, in their setup the sequential component was taken over multiple data points, which assumes sequential dependency among data samples in the classification task.
The RL framework CMAB offers a promising alternative because it considers the input as a sequence of independent states. We formalize our classification task as a CMAB problem, because this is consistent with the independence of data points in the commonly shuffled datasets. Dudík et al. (2014) use CMAB agents by modifying K-class classification as a K-armed bandit problem, where the agent receives a reward of 1 for correct and 0 for incorrect classification. Dimakopoulou et al. (2019) use this framework and modify different CMAB algorithms to balance exploration and exploitation and compare the original and modified agents on 300 classification datasets. However, their analysis focused on datasets with either limited classes, features, or observations. To the best of our knowledge, we are the first to extend reward scaling for fair multi-class classification or to apply reward scaling for classification with CMAB.
3 Methodology
In this section, we describe how we formalize our classification task as a CMAB. We introduce three RL methods and explain how we adapt them for fair classification.222Due to space limitations, we only summarize the key idea of the algorithms and how we adapt them in the paper. Please refer to the Appendix and original papers for more details.
3.1 Contextual Multi-Armed Bandit
We formalize the multi-class classification task as a finite contextual multi-armed bandit (CMAB) problem. In each round , an agent is presented with a context vector . The agent chooses an action from a fixed set of arms, based on the policy . After the action is taken, the environment returns a reward: . In a multi-class classification framework, the action space is the set of all possible classes, while the context vector is a representation of the input, e.g. a contextual text embedding (see Section 4.1 for more information). Within a finite number of rounds, the agent aims to learn the optimal policy to maximize the total reward. In other words, given a set of testing data, we aim to learn the optimal policy to maximize the selection of correct classes.
We extend the CMAB framework for fair classification by constructing a reward function that counters data imbalances. We assign a reward scale for each sensitive state , comprising the desired class (e.g. occupation) and protected attribute (e.g. gender). The total reward for a given prediction is calculated as . It comprises an accuracy term , and a reward scale matrix . Unlike previous work Dudík et al. (2014), which defines the accuracy term as , we define it as . Which allows us to scale the reward for both correct (+1) and incorrect classifications (-1). We use the term reward scale to indicate that this approach adjusts the magnitude but not the sign of the reward. Section 3.3 presents various designs of the reward scale.
3.2 Reinforcement Learning Algorithms
We select three different RL algorithms and adapt them to learn optimal policies for fair classification in the formalized CMAB problem. These algorithms include one classical CMAB algorithm that addresses the linear relationship between the expected reward and the context, as well as two popular deep RL algorithms for MDP problems, Deep Q-Network (DQN) and Proximal Policy Optimization (PPO), which allow us to leverage non-linear approximations. The two deep RL algorithms are selected as they are representative of the two key types of deep RL approaches: value-based methods and policy gradient methods. By employing these three algorithms, we aim to investigate the application of diverse RL methods.
3.2.1 LinUCB
The classical CMAB algorithm, disjoint Linear UCB (LinUCB) Li et al. (2010) assumes a linear relationship between the context embedding and the reward . A benefit of disjoint LinUCB over other CMAB algorithms is that each class has a unique learnable weight vector , which makes it suitable for classification with many classes. In each round, the agent chooses the arm (i.e. class label) with the highest score , based on the context vector . This is a combination of the mean of the expected payoff, , and the standard deviation , weighted with parameter to control the level of exploration. The weight vector of each arm is defined as . Here the covariant matrix is calculated with the history of context vectors chosen by that arm, . The vector is the mean context vector of the arm weighted by the obtained rewards, .
3.2.2
To adapt the MDP algorithms for a CMAB problem, our CMAB implementation is congruent with a one-step MDP, where each initial state is sampled from the existing set of context , and each second state is a terminal state. In DQN Mnih et al. (2015), the agent learns a Q-function, parameterized by , to estimate the return for each state-action pair. According to the Bellman equation Bellman (1957), the optimal Q-value, , of two sequential states are linked by:
(1) |
In our case (the one-step MDP), each next state is the terminal state, after which there is no reward, thus we obtain, , and . The parameters of are optimized using the mean-squared error between the current Q-value, , and the updated value provided in Equation 1. The updated value is computed as , but since the next state is always the terminal state it reduces to . We finalize the adaptation of DQN for the CMAB by casting the states as context vectors, obtaining the loss function:
The network is updated by sampling a minibatch of tuples from the replay buffer. The enables exploration using an -greedy policy for selecting actions.
3.2.3
Different from DQN, in Proximal Policy Optimization (PPO) Schulman et al. (2017), the policy (parameterized by ) is directly optimized under the objective of selecting the best action. The general objective in policy gradient methods is to maximize: . The advantage is computed as , where a critic network is used to estimate the state value. PPO ensures the policy does not deviate too far during an update, by scaling the advantage with the probability ratio, . This ratio is clipped to create a conservative lower bound to control the policy’s change at each step. The actor’s objective function is thus defined as:
To adapt PPO for CMAB, the sequential component is removed and the state is replaced by the context vector . For the actor loss, the advantage changes and is calculated as . The return again reduces to the reward, thereby simplifying the critic loss to:
Lastly, the final loss of the agent contains a penalty that maximizes the policy’s entropy of the context vector to encourage exploration.
3.3 Reward Scales
Lin et al. (2020) implement reward scaling on an imbalanced binary classification dataset. Inspired by this, we propose different ways of reward scales for multi-class classification with imbalances of protected attributes. For context, we use the profession classification dataset, BiasBios, where reward scaling tackles the sub-class imbalance of the protected attribute gender. To illustrate the influence of various reward scales Figure 2 shows the scales of a balanced (Professor) and an imbalanced (Nurse) class for the protected groups with attribute gender.
For the first method, we extend the implementation of Lin et al. (2020) into the multi-classification setting and reduce the reward for the majority by scaling it with the imbalanced ratio , which is the ratio between the number of samples of the minority and majority class in class .
Figure 2 demonstrates that scales with a reverse of the bias within a class, however, compared to a balanced class, the reward scale of the majority is very low. Therefore, we propose a second design that keeps the scales of the majority group in the imbalanced class equal to the scales of the balanced class. Thus we set the majority value at 1 and only increase the minority value, based on the inversed imbalanced ratio.
The third implementation adopts the Equal Opportunity (EO) formalization used by Han et al. (2022a). Contrary to the previous two methods it ensures the average weights per class remain equal, providing an improved theoretical fairness among classes. The EO objective is achieved by aggregating the loss per sensitive state and then scaling it. However, our work scales per instance, thus we convert the EO objective to instance-specific weights (see Appendix B.3) and obtain:
Lastly, we also employ the Inverse Probability Weighting (IPW) technique Höfler et al. (2005). Full fairness across classes and protected groups is obtained by scaling with the joint probability, resulting in:
![Refer to caption](x2.png)
3.4 Supervised Learning: Loss reweighting
Parallel to reward scaling in RL is (instance) reweighing in supervised learning Han et al. (2022a); Lahoti et al. (2020), here loss reweighting for clarity. Loss reweighting has been a popular technique for imbalanced datasets, where the loss of each data sample is scaled to mitigate the class imbalance, traditionally using the IPW Höfler et al. (2005). The weighted cross-entropy loss using the true probability , predicted probability :
We implement supervised learning with loss reweighing to serve as a strong baseline and highlight the connection between loss reweighing and reward scaling.
4 Experiments
4.1 Dataset
The BiasBios De-Arteaga et al. (2019) consists of 393,423 biographies labeled with one of 28 professions, and a binary gender label. Following De-Arteaga et al. (2019), the data is randomly split according to 65% training, 25% testing, and 10% for validating. The dataset contains two imbalances: varying frequencies of the professions and a difference in gender percentage for each class.
Following Ravfogel et al. (2020) and Han et al. (2022a) we also evaluate on the Emoji (Elazar and Goldberg, 2018) sentiment analysis task of Twitter data Blodgett et al. (2016). The task involves binary sentiment classification evaluation with race as the protected attribute, approximated through the provided labels Standard American English (SAE) and African American English (AAE). As per Han et al. (2021), the dataset is composed of Happy (40% AAE, 10% SAE), and Sad: (10% AAE, 40% SAE). We use the same train, dev, and test splits of 100k/8k/8k instances, respectively.
Context Vectors
Each textual data sample is embedded into a context vector via a pretrained encoder, enabling the algorithms for classification. Following Ravfogel et al. (2020) we use the same fixed pretrained encoder for each dataset. For the BiasBios dataset, each biography is encoded using the [CLS] output of the uncased BERT-base model Devlin et al. (2019). For the Emoji dataset, we use the DeepMoji encoder Felbo et al. (2017), which has been demonstrated to capture a diverse range of moods and demographic information.
4.2 Metrics
Following prior work, we evaluate performance using accuracy and fairness using the True Positive Ratio (TPR) gap De-Arteaga et al. (2019); Ravfogel et al. (2020). The TPR gap of a class is calculated as: , where and represent the two options for the sensitive states. The global TPR metric, GAP, is then calculated as the root mean square of the individual metrics:
(2) |
To quantify performance and fairness as a single metric we use the Distance To the Optimum (DTO) introduced in Han et al. (2022a). DTO combines the metrics (accuracy, 1-GAP) as dimensions of evaluation space and computes the Euclidean distance between the achieved and Utopian point. The smaller the distance to the Utopian point (lower DTO), the better. We report the DTO with the Utopian accuracy and GAP as the best values across all evaluated models.
While accuracy measures the overall performance and GAP the disparity among protected groups within a class, these metrics do not capture imbalance performance across classes. Therefore we also evaluate our algorithms using the macro-averaged F1 metric to detect if minority classes are ignored. All metrics are scaled by 100 for ease of reading and all metrics are represented in the tables as the mean std over 5 random seeds, except DTO which is taken over the mean.
4.3 Hyperparameters and model selection
Each algorithm uses the same classifier architecture, except LinUCB, which has a custom set of learnable parameters. The classifier has one hidden layer MLP. All models are trained for 10 epochs, except LinUCB, which achieved optimal performance within 2 epochs. All models are evaluated on the validation set after 50k iterations to account for different convergence speeds of models. The best model throughout training and across hyperparameters is selected using DTO. We apply hyperparameter optimization on both datasets for each of the algorithms, for details see Appendix A.3.
PPObandit | Sup | |||
---|---|---|---|---|
Accuracy | GAP | Accuracy | GAP | |
74.6 0.7 | 9.9 0.8 | 79.3 0.1 | 7.9 0.3 | |
78.8 0.1 | 8.4 0.6 | 79.8 0.3 | 6.9 0.2 | |
79.2 0.2 | 8.5 0.2 | 80.1 0.2 | 7.1 0.5 | |
45.8 6.9 | 10.5 0.9 | 72.1 0.7 | 6.1 0.3 |
BiasBios (28 Classes) | Emoji (2 Classes) | ||||||||
---|---|---|---|---|---|---|---|---|---|
Algorithm | Accuracy | GAP | DTO | F1 | Zeit | Accuracy | GAP | DTO | Zeit |
Sup | 81.0 0.1 | 16.4 0.5 | 9.3 | 73.8 0.3 | 1.0 | 72.3 0.1 | 38.1 0.6 | 28.3 | 1.0 |
INLP | 80.2 0.6 | 9.7 0.4 | 2.8 | 71.7 1.4 | 50.1 | 63.5 3.6 | 24.1 5.4 | 18.6 | 3.6 |
MP | 81.1 0.1 | 13.9 0.6 | 6.8 | 74.0 0.2 | 2.6 | 71.8 0.3 | 17.1 1.0 | 8.1 | 2.3 |
BTEO | 79.2 0.3 | 8.4 0.6 | 2.3 | 68.1 0.4 | 1.7 | 75.4 0.1 | 10.4 1.0 | 0.4 | 0.8 |
DAdv | 80.8 0.2 | 8.5 0.6 | 1.4 | 72.9 0.4 | 4.8 | 75.6 0.3 | 11.6 1.7 | 1.6 | 5.7 |
SupEO | 80.1 0.2 | 7.1 0.5 | 1.0 | 71.7 0.5 | 1.0 | 75.5 0.1 | 11.4 1.1 | 1.4 | 1.0 |
LinUCBEO | 74.6 0.2 | 12.2 0.5 | 8.3 | 59.8 1.1 | 31.9 | 75.3 0.2 | 10.4 0.7 | 0.5 | 2.8 |
79.2 0.1 | 10.1 0.4 | 3.6 | 66.4 0.2 | 57.4 | 70.8 0.8 | 10.0 1.0 | 4.8 | 30.2 | |
79.2 0.2 | 8.5 0.2 | 2.4 | 66.0 0.8 | 2.9 | 75.4 0.1 | 14.4 0.6 | 4.4 | 3.0 |
4.4 Comparison Models
Besides the supervised implementation in Section 3.4, abbreviated to Sup, we also compare our models against various existing debiasing methods. INLP Ravfogel et al. (2020) debiases embeddings by iteratively training classifiers to predict the protected attribute, it then removes this information from the embedding using a projection of the classifier’s nullspace. MP Haghighatkhah et al. (2022) simplifies the INLP setup by using a single Mean Projection (MP) between the representation of each class’s protected groups. DAdv, Han et al. (2021) removes sensitive information from the embeddings by applying adversarial training using diverse adversaries. Lastly, BTEO Han et al. (2022a) subsamples the dataset to establish equal opportunity. We implement these existing methods with the same training settings for fair comparison. Notably, we highlight how Supervised is theoretically equal to instance reweighing in Han et al. (2022a), but our implementation achieves significantly higher performance.
5 Results and Analysis
We train each of the RL algorithms with and without reward scaling. As a strong baseline, we also train a supervised learning model in the standard fashion and use loss reweighting with the same reward scale function.
5.1 Different Reward Functions
We first evaluate the effect of the different reward scales discussed in Section 3.3 by providing the results for the implementations of PPO and Sup, see Table 1, for full table see Appendix D.4.
The results presented in the table demonstrate that the imbalance ratio yields substantial gains in fairness and accuracy when applied to increase the reward for the minority class () as opposed to decreasing the reward for the majority class (). Especially the accuracy of PPO is sensitive to this, suggesting that PPO might not work very well for states with low reward scales.
As hypothesized, scaling with the joint probability of class and protected attribute as in proves to be too unstable. It results in the worst accuracy for both algorithms, with a minor improvement in fairness for supervised learning. The overall difference between and is minimal as expected from their similar reward scales depicted in Figure 2. We use EO in our experiments because of its better theoretical foundation.
5.2 Main Results
The main results of our experiments are summarized in Table 2.
On the BiasBios dataset, our DQN and PPO implementations achieve strong results, with PPO outperforming DQN in fairness, as PPO’s lower GAP shows. Our supervised implementation, Sup , surpasses all other baselines. LinUCB performs significantly worse on this task, obtaining one of the worst DTO scores. In contrast, on the Emoji dataset, LinUCB achieves one of the best performance-fairness trade-offs, as indicated by the low DTO score, and is only surpassed by BTEO, which has a slightly higher accuracy. On the other hand, DQN obtains a lower accuracy and PPO lower fairness compared to other metrics.
These results suggest that the classical CMAB algorithm LinUCB excels on binary datasets, while the deep RL implementations, DQN and PPO, perform better in multi-class settings. Notably, although DQN and PPO obtain competitive results on the BiasBios, their F1 score is considerably lower than the baselines. Further analysis of per class metrics (see Appendix D.2) reveals that while the F1 for most classes was on par with the supervised setup, both deep RL algorithms failed to recall two of the very sparse classes.
Compared to baseline methods such as BTEO and DAdv, our DQN and PPO implementations demonstrate competitive performance on the BiasBios dataset. Table 2 also shows that, contrary to previous work Han et al. (2022a),333The EO scaled supervised implementation of Han et al. (2022a) achieves an Accuracy of 75.7 and GAP of 13.9 loss-scaling for supervised learning (Sup ) achieves superior overall performance to downsampling (BTEO). Downsampling only seems to outperform scaling when enough data is present, as demonstrated by its lower GAP for the Emoji dataset.
Algo + | Accuracy | GAP | F1 |
---|---|---|---|
Sup | 81.0 (- 0.9) | 16.4 (- 9.3) | 73.8 (- 2.1) |
LinUCB | 78.4 (- 3.8) | 15.5 (- 3.3) | 67.3 (- 7.5) |
80.1 (- 0.9) | 13.7 (- 3.6) | 66.5 (- 0.1) | |
79.7 (- 0.5) | 14.4 (- 5.9) | 67.5 (- 1.5) |
5.3 Reward Scaling Impact
We investigate the influence of reward scaling on our models by training them with and without scaling. Table 3 presents the results on BiasBios as the mean performance without scaling and the change in metrics when EO scaling is applied.
Without reward scaling the three RL algorithms achieve similar accuracy to the supervised approach but at the cost of a lower F1 score. As mentioned above, the RL algorithms fail on two very sparse classes, which explains the drop in GAP and F1. Failing to classify any instances of a class correctly results in a TPR gap of 0 for that class, since the result is "fair" among both genders.
The EO reward scale significantly reduces the GAP of all implementations, at the cost of a slight decrease in Accuracy and F1 for most models. However, on LinUCB the scaling causes a large performance reduction with only a small GAP reduction, suggesting that scaling hinders the performance more than it improves the fairness.
To inspect the weak effect of reward scaling on LinUCB, we analyze the TPR gap per profession against the gender imbalance for both cases in Figure 3. Without scaling, LinUCB’s performance follows a predictable positive correlation with gender imbalance. For instance, in the Nurse class ( 90% women), the model performs better for the majority group (women), resulting in a positive TPR gap. However, after reward scaling this correlation is inverted, causing the model to perform worse for the majority group and better for the minority group. In case of the Nurse class, the model obtains a negative TPR gap. This suggests LinUCB is oversensitive to scaling on the BiasBios dataset, causing it to overcompensate and penalize the majority group.
![Refer to caption](x3.png)
![Refer to caption](x4.png)
5.4 Subclass Imbalance Sensitivity
For a more in-depth analysis of each model’s sensitivity to subclass imbalance, we train them on the Emoji dataset under various stereotyping ratios. A stereotyping ratio represents the proportion of the AAE and SAE samples in each class. For example, a stereotyping ratio of 0.2 means the data is distributed as Happy (20% AAE, 80% SAE), Sad (80% AAE, 20% SAE).
![Refer to caption](x5.png)
Figure 4 reveals a strong inverse relationship between LinUCB’s fairness and the stereotyping ratio. Although the stereotypical ratios are symmetric at the value of 0.5 the fairness of LinUCB is asymmetric at this value. Thus there is a residual representation bias in the data that is not addressed by the reward scaling. In contrast, supervised learning maintains a mostly stable fairness, except for the most extreme ratios. Interestingly, LinUCB reveals a reverse pattern in best and worst fairness.
The relatively low accuracy of DQN and poor performance on fairness of PPO are consistent across ratios. However, PPO does have the most constant fairness and performance across stereotyping ratios, indicating good training stability.
5.5 Signal strength vs. Scaling
We now examine how the strength of the protected information affects the efficacy of reward scaling. We focus on two scenarios that modify the gender signal in the representations: 1) adding explicit gender information, thus increasing the gender signal strength 2) debiasing the embeddings using MP, which reduces it. Table 4 presents the mean and relative difference compared to the results without the specified modification.444Full tables in Appendix D.5
Providing the model with gender information increases the overall accuracy. However, the impact on fairness, as indicated by the GAP score varies among algorithms. The GAP score increases for the two algorithms with the lowest GAPs (Sup, PPO) and decreases for the two with the highest GAP (LinUCB, DQN). Only the algorithms that perform less well on fairness benefited from access to protected attribute.
Explicit Gender Info | MP-Debiased | |||
---|---|---|---|---|
Algo+ | Accuracy | GAP | Accuracy | GAP |
Sup | 80.2 (+ 0.1) | 7.2 (+ 0.1) | 80.0 (- 0.1) | 7.4 (+ 0.3) |
LinUCB | 74.5 (+ 0.1) | 11.7 (- 0.5) | 74.3 (- 0.3) | 11.5 (- 0.7) |
79.2 (+ 0.0) | 10.0 (- 0.1) | 79.0 (- 0.2) | 8.6 (- 1.5) | |
79.3 (+ 0.1) | 8.7 (+ 0.2) | 79.2 (+ 0.0) | 9.7 (+ 1.2) |
Removing the bias with MP reduces the test accuracy for nearly all algorithms, indicating some useful information is removed. Again, the modification increased relatively low GAP scores, and decreased relatively high scores. As such, changing to a representation with relatively low bias helps LinUCB and DQN, whereas Sup and PPO that already achieved better fairness mainly see their overall performance hindered.
Notably, the differences in Table 4 are relatively small and hardly ever surpass the standard deviation provided in Table 2. This suggests that while the strength of the protected information influences performance and fairness, the impact might be less pronounced than the choice algorithmic design. Moreover, all methods reduced the GAP score compared to only applying MP (which yields a GAP of 13.9, see Table 2).
6 Discussion and Conclusion
This paper introduces a novel approach to fair classification using the Contextual Multi-Armed Bandit (CMAB) framework and explores various Reinforcement Learning (RL) algorithms. Our findings demonstrate the potential of different RL algorithms for this task and the efficacy of reward scaling in mitigating imbalances of protected groups. The results show the MDP-derived deep RL methods perform best on the multi-class dataset, while the classical bandit algorithm, LinUCB, excels on the binary dataset. Moreover, our scaled supervised learning implementation achieved new state-of-the-art results on the complex BiasBios dataset.
Our experiments also revealed two limitations 1) RL algorithms can ignore some very sparse classes, despite performing well under most class imbalances. 2) Reward scaling for LinUCB can impair majority group performance beyond that of the minority group. However, the effect of reward scaling remains robust across varying strengths of the protected information, highlighting its potential as a powerful tool for achieving fair outcomes.
Despite these challenges, we believe that the proposed framework presents a promising approach to leverage RL algorithms for fair classification, opening up new research avenues. We encourage future work to extend upon our framework by exploiting different RL characteristics , such as model updates for MDP algorithms based on non-differentiable fairness metrics.
Limitations
Important limitations of this work can be divided into two sections: 1) Limitations of the dataset and data requirements of our models 2) Limitations specific to our algorithms and experiments, independent of the data.
Data limitation
Firstly, all datasets considered in this study used English text, which restricts the analysis and might miss other types of biases related to different linguistic and cultural contexts. Secondly, the protected groups evaluated in this study simplified to binary labels, which excludes people who do not fall into this category such as non-binary individuals and the multidimensional nature of ethnicity.
Our reward scaling approach also requires these labels for classification. Although our setup could easily be extended to cases with more labels, it would be interesting to see fair classification with protected attributes as continuous values. But due to lack of good benchmarks restricts the evaluation of such cases.
Algorithmic limitation
Firstly, our paper used two deep RL MDP algorithms and one linear classical CMAB agent.
We recognize that while linear agents have a significant focus in the CMAB literature, the fast field of CMAB agents includes options with non-linear algorithms that could also be applied to this task.
The choice of LinUCB does not represent the state-of-the-art, but rather a classical high-performance implementation.
Second, the various hyperparameters limit the extent of general statements about each algorithm.
We have documented our hyperparameter search and training methods in the appendix, to ensure the interpretability of our experiments, but our results only demonstrate the capabilities of our best implementation.
Moreover, the use of DTO to select the best model throughout training fails to account for potential trade-offs between fairness and accuracy at different points in training. For example, on the Emoji dataset, PPO underperformed in Fairness and DQN in accuracy.
However, it is possible that at another pointing training with a higher DTO score, the trade-off between fairness and accuracy was reversed.
Ethics Statement
The application of the paper was to improve fairness among protected groups in classification. However, no algorithm is able to obtain perfect fairness and remove the bias perfectly. Therefore applications of the mentioned debiasing methods should always strongly take the mentioned limitations into account. Moreover, the current experiments are limited to specific datasets and real world use cases may be different. Careful evaluation and testing system behavior in the intended setting with input from experts who can judge the consequences of remaining bias is essential.
Acknowledgments
This research was partially funded by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https://hybrid-intelligence-centre.nl.
References
- Aguirre et al. (2023) Carlos Aguirre, Kuleen Sasse, Isabel Cachola, and Mark Dredze. 2023. Selecting shots for demographic fairness in few-shot learning with large language models. arXiv preprint arXiv:2311.08472.
- Bellman (1957) Richard Bellman. 1957. Dynamic Programming, 1 edition. Princeton University Press, Princeton, NJ, USA.
- Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic Dialectal Variation in Social Media: A Case Study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130.
- De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120–128.
- Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
- Dimakopoulou et al. (2019) Maria Dimakopoulou, Zhengyuan Zhou, Susan Athey, and Guido Imbens. 2019. Balanced linear contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3445–3453.
- Dudík et al. (2014) Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485–511.
- Elazar and Goldberg (2018) Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11–21.
- Felbo et al. (2017) Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625.
- Haghighatkhah et al. (2022) Pantea Haghighatkhah, Antske Fokkens, Pia Sommerauer, Bettina Speckmann, and Kevin Verbeek. 2022. Better hit the nail on the head than beat around the bush: Removing protected attributes with a single projection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8395–8416.
- Han et al. (2021) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2021. Diverse adversaries for mitigating bias in training. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2760–2765.
- Han et al. (2022a) Xudong Han, Timothy Baldwin, and Trevor Cohn. 2022a. Balancing out bias: Achieving fairness through balanced training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11335–11350.
- Han et al. (2022b) Xudong Han, Aili Shen, Yitong Li, Lea Frermann, Timothy Baldwin, and Trevor Cohn. 2022b. Fairlib: A unified framework for assessing and improving fairness. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 60–71.
- Höfler et al. (2005) Michael Höfler, Hildegard Pfister, Roselind Lieb, and Hans-Ulrich Wittchen. 2005. The use of weights to account for non-response and drop-out. Social psychiatry and psychiatric epidemiology, 40:291–299.
- Lahoti et al. (2020) Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. 2020. Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems, 33:728–740.
- Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661–670.
- Lin et al. (2020) Enlu Lin, Qiong Chen, and Xiaoming Qi. 2020. Deep reinforcement learning for imbalanced classification. Applied Intelligence, 50:2488–2502.
- Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1–35.
- Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540):529–533.
- Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256.
- Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
- Shen et al. (2022) Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2022. Does representational fairness imply empirical fairness? In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 81–95.
- Sherman and Morrison (1950) Jack Sherman and Winifred J Morrison. 1950. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 21(1):124–127.
- Subramanian et al. (2021) Shivashankar Subramanian, Afshin Rahimi, Timothy Baldwin, Trevor Cohn, and Lea Frermann. 2021. Fairness-aware class imbalanced learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2045–2051.
- Uc-Cetina et al. (2023) Victor Uc-Cetina, Nicolas Navarro-Guerrero, Anabel Martin-Gonzalez, Cornelius Weber, and Stefan Wermter. 2023. Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56(2):1543–1575.
- Wadsworth et al. (2018) Christina Wadsworth, Francesca Vera, and Chris Piech. 2018. Achieving fairness through adversarial learning: an application to recidivism prediction. In Proceedings of the FAT/ML Workshop.
- Wang et al. (2019) Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5310–5319.
- Wiering et al. (2011) Marco A Wiering, Hado Van Hasselt, Auke-Dirk Pietersma, and Lambert Schomaker. 2011. Reinforcement learning algorithms for solving classification problems. In 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 91–96. IEEE.
Appendix A Reproducibility
A.1 Data Analysis
Because the BiasBios dataset needs to be scraped online, we provide the full composition of the BiasBios dataset split up in profession and gender in Table 5.
Profession | Female | Male |
---|---|---|
Professor | 53290 | 64820 |
Physician | 19579 | 18986 |
Attorney | 12494 | 20113 |
Photographer | 8689 | 15635 |
Journalist | 9873 | 10077 |
Nurse | 17236 | 1735 |
Psychologist | 11385 | 6910 |
Teacher | 9768 | 6428 |
Dentist | 5153 | 9326 |
Surgeon | 1972 | 11301 |
Architect | 2398 | 7715 |
Painter | 3543 | 4193 |
Model | 6214 | 1288 |
Poet | 3441 | 3570 |
Filmmaker | 2310 | 4699 |
Software Engineer | 1089 | 5817 |
Accountant | 2081 | 3571 |
Composer | 918 | 4682 |
Dietitian | 3689 | 289 |
Comedian | 592 | 2207 |
Chiropractor | 690 | 1908 |
Pastor | 609 | 1923 |
Paralegal | 1499 | 268 |
Yoga Teacher | 1406 | 257 |
Dj | 211 | 1274 |
Interior Designer | 1183 | 280 |
Personal Trainer | 654 | 778 |
Rapper | 136 | 1271 |
rapper | 136 | 1271 |
A.2 Model Selection
Selecting the best model throughout training or across hyperparameters is strongly dependent on the selection metric. To balance fairness and performance we use the proposed method of Han et al. (2022a), and select using DTO. The full equation of DTO is provided below, where the obtained metrics are determined by the point , and the utopian metrics are .
The best training timestep according to DTO is determined with utopian values (1,1), and the best hyperparameters setting utopian values as the best metric values during training (i.e. the highest performance and fairness each individually obtained, which do not necessarily belonging to the same algorithm).
The reported DTO values in table X and Y are obtained using the best performance and accuracy method as: [performance, fairness] BiasBios 28C = [0.811, 0.929], BiasBios 8C = [0.868, 0.978], Moji=[0.756, 0.900]
A.3 Hyperparameters
The architecture of the neural network for each algorithm is fixed and consists of 2 layers MLP. For the critic in PPO the architecture is the same except for the output size which is 1. Hyperparameter optimization is applied for each of the parameters of the algorithms using grid search. Table 7 shows the ranges and the best values.
Typ | Dimensions | |
---|---|---|
Layer 1 | Linear | |
Layer 2 | Linear | |
Activation | ReLU | |
Optimizer | Adam |
Algorithm | Parameter | Min | Max | Best | |
---|---|---|---|---|---|
BiasBios | Emoji | ||||
lr (actor) | |||||
lr (critic) | |||||
PPO | Batch size | 64 | 512 | 512 | 512 |
Entropy | 0.01 | 0.1 | 0.2 | 0.1 | |
-clip | 0.05 | 0.3 | 0.1 | 0.3 | |
Supervised | lr | ||||
Batch size | 64 | 512 | 128 | 512 | |
lr | |||||
DQN | Batch size | 32 | 256 | 256 | 32 |
Eps_end | 0.001 | 0.1 | 0.1 | 0.01 | |
Eps decay | 0.5 | 1.0 | 0.5 | 0.5 | |
LinUCB | 0.1 | 3.0 | 1.5 | 2.5 |
Related work implementations
Following previous work Ravfogel et al. (2020); Han et al. (2022b), we use INLP and MP in a post hoc manner to the features extracted from the last hidden layer of the supervised model and train a logistic classifier for the final classification. For our MP debiasing experiments in section 5.5 we use MP to debias the context vectors before training, instead of poshoc on the hidden layer of the trained network.
Appendix B Algorithms
B.1 Single-Step Markov Decision Process
To formalize how the policy-gradient methods such as PPO relate to the Contextual Multi-Armed Bandit framework, we define below the single-step Markov Decision Process. An MDP is defined by the tuple , and our single-step variant contains only two states . The initial state is sampled each time from the environment and for our classification setup is part of the set of context embeddings, . To ensure data samples are treated independently the second state is always the terminal state . The action space is equal to the number of classes: . The reward function is equal to that of the CMAB and is defined in section 3.1. Lastly, each trajectory is defined as and both the transition probability, , and the discount factor are irrelevant since each action results in the terminal state.
B.2 LinUCB
The full algorithm of LinUCB from Li et al. (2010), used in the paper is shown in Algorithm
B.3 Equal Opportunity Weights
Where Han et al. (2022a) used EO for supervised learning, their implementation achieved this objective by grouping the loss per class and then averaging over them. In this section, we see how we can use this to obtain the weights for each data sample based on the class and protected attribute . For two protected groups and in class , let and be the number of samples for and , and and , be the weights. To get a statement of the weights with EO for each sensitive state, , we need two axioms.
Axiom 1. The weight scale ratio between the two protected groups of a class should be inversely proportional to their probability in the dataset:
Axiom 2: To ensure fairness across classes, the average weight per profession should be a fixed value so that:
Combining these two axioms we obtain the formulation:
For the multi-class classification task the average reward scale, , should be 1, and the probability is conditional on the class , obtaining the final equation:
Appendix C Ablation Experiments
Here we add our experiments that did not make the main paper.
C.1 Analysis: Model and Data Efficiency
An important aspect for evaluation is related to the data and computational of each algorithm. For ease of comparison, all algorithms except LinUCB were trained for 10 epochs. However, DQN and PPO each reuse the seen data in a different way to deal with the data sparsity of standard RL settings. DQN is updated using a replay-buffer from which it samples a minibatch of N triplets for each iteration. In contrast, PPO collects N samples during the observation phase after which it updates the model with this batch number of times. Lastly, LinUCB achieves optimal results after 1 epoch but is constrained by the computations of its weight matrices, which require the inverse of a square matrix with dimension . For computational efficiency, we use the Sherman–Morrison formula which updates the previous computed inverse with a rank one update Sherman and Morrison (1950)
The time complexities in Table 2, demonstrate that PPO is closest to supervised learning and that DQN takes significantly more time since it needs to sample from the buffer at each iteration.
Notably, LinUCB is strongly dependent on the number of classes, reducing its relative efficiency from 32 to 3 times that of Supervised Learning. The bottleneck here is that it needs to compute an upper confidence bound for each class.
Another important feature is the sensitivity to hyperparameters. PPO and DQN are sensitive to several hyperparameters that determine the level of its exploration, such as DQN’s mini-batch size or exploration parameter, or PPO’s entropy and clipping coefficients.
LinUCB is easiest to implement in this regard and does not require any neural network hyperparameters, but only one exploration parameter , see section B.2.
Appendix D Full result for experiments
To distinguish the sensitivity of gender imbalance and data-sparsity we also run experiments with a subset of the data, following Aguirre et al. (2023), and select only the professions that have at least 1000 samples for both genders in the test set, resulting in 8 professions.
D.1 BiasBios: training performance over time
In Reinforcement Learning literature it is common to provide the performance of an algorithm throughout training for evaluation. Therefore we provide the evaluation accuracy of our four algorithms in Figure 5
![Refer to caption](x6.png)
![Refer to caption](x7.png)
D.2 BiasBios: Recall per profession
As a further analysis of the lacking F1 score of the RL algorithms compared to the supervised implementation, we provide the Recall scores as a percentage of the class. Since class 21, Professor appears significantly more often than the most common class after it, we leave it out for clarity.
![Refer to caption](extracted/5720419/images/recall_per_prof/recall_dqn.png)
![Refer to caption](extracted/5720419/images/recall_per_prof/recall_linucb.png)
![Refer to caption](extracted/5720419/images/recall_per_prof/recall_ppo.png)
![Refer to caption](extracted/5720419/images/recall_per_prof/recall_sup.png)
D.3 Full tables: BiasBios (28C and 8C)
Some of our results in section 5.3 are presented as the mean only. The full results of our algorithms as the mean and std over the five seeds is provided in the tables here. Table 8 shows the performance of our algorithms with and without reward scaling on the BiasBios dataset with the 28 and 8 classes.
28 Classes | 8 Classes | |||||||
---|---|---|---|---|---|---|---|---|
Algorithm | Accuracy | GAP | DTO | F1 | Accuracy | GAP | DTO | F1 |
Sup | 81.0 0.1 | 16.4 0.5 | 10.0 | 73.8 0.3 | 86.8 0.1 | 8.3 0.7 | 6.2 | 82.7 0.1 |
LinUCB | 78.4 0.1 | 15.5 0.3 | 9.6 | 67.3 0.4 | 85.3 0.2 | 7.6 0.3 | 5.8 | 80.6 0.2 |
80.1 0.2 | 13.7 0.3 | 7.2 | 66.5 1.3 | 86.5 0.2 | 7.6 0.3 | 5.5 | 82.2 0.2 | |
79.7 0.5 | 14.4 0.7 | 8.0 | 67.5 2.0 | 86.0 0.2 | 8.7 0.4 | 6.7 | 81.6 0.2 | |
SupEO | 80.1 0.2 | 7.1 0.5 | 1.1 | 71.7 0.5 | 86.3 0.2 | 2.4 0.1 | 0.6 | 82.0 0.2 |
LinUCBEO | 74.6 0.2 | 12.2 0.5 | 9.6 | 59.8 1.1 | 83.4 0.2 | 7.6 0.3 | 6.8 | 77.6 0.3 |
79.2 0.1 | 10.1 0.4 | 3.9 | 66.4 0.2 | 86.2 0.1 | 2.2 0.2 | 0.7 | 81.6 0.2 | |
79.2 0.2 | 8.5 0.2 | 2.7 | 66.0 0.8 | 85.8 0.1 | 2.8 0.6 | 1.3 | 81.4 0.2 |
D.4 Full tables: four reward scaling methods
The results from reward scaling using the four described scales and our four algorithms are shown in Table 9.
Algo | Accuracy | GAP | F1 | |
---|---|---|---|---|
SUP | 79.3 0.1 | 7.9 0.3 | 69.3 0.3 | |
79.8 0.3 | 6.9 0.2 | 71.8 0.6 | ||
80.1 0.2 | 7.1 0.5 | 71.7 0.5 | ||
72.1 0.7 | 6.1 0.3 | 64.8 0.8 | ||
PPO | 74.6 0.7 | 9.9 0.8 | 49.7 2.2 | |
78.8 0.1 | 8.4 0.6 | 64.7 0.8 | ||
79.2 0.2 | 8.5 0.2 | 66.0 0.8 | ||
45.8 6.9 | 10.5 0.9 | 45.3 5.8 | ||
DQN | 76.2 1.1 | 10.4 0.7 | 57.2 4.8 | |
79.3 0.1 | 11.1 0.6 | 65.8 1.4 | ||
79.2 0.1 | 10.1 0.4 | 66.4 0.2 | ||
74.6 0.3 | 12.8 0.2 | 56.6 0.3 | ||
LinUCB | 72.8 0.1 | 12.0 0.5 | 54.6 0.9 | |
74.1 0.4 | 11.6 0.5 | 59.3 1.7 | ||
74.6 0.2 | 12.2 0.5 | 59.8 1.1 | ||
37.3 2.5 | 10.3 0.7 | 35.4 1.0 |
D.5 Full results: Explicit gender information and Ensemble techniques
This section includes the full results of Section 5.5, after adding the gender information explicitely and after removing it with MP. The results are presented as mean and standard deviation over 5 seeds in Table 10 and Table 11
Algo + g | Accuracy | GAP | F1 |
---|---|---|---|
80.2 0.2 | 7.2 0.5 | 71.9 0.7 | |
74.5 0.2 | 11.7 0.5 | 59.6 0.8 | |
79.2 0.2 | 10.0 0.5 | 66.1 0.4 | |
79.3 0.1 | 8.7 0.3 | 66.1 0.6 |
Algo + MP | Accuracy | GAP | F1 |
---|---|---|---|
80.0 0.2 | 7.4 0.4 | 71.9 0.3 | |
74.3 0.4 | 11.5 0.1 | 59.4 1.0 | |
79.0 0.2 | 8.6 0.3 | 65.8 0.6 | |
79.2 0.2 | 9.7 0.6 | 66.8 1.4 |