A Novel Lexicon for the Moral Foundation of Liberty

Based on the approach proposed by Turney and Littman (2003), our first strategy for generating lexicons relies on word embedding similarity between the vectors of the positive and negative instances of a dataset’s documents. Hence, the method relies on a set of seed words that accurately represent the domains we aim to differentiate. However, arbitrary selection of seed words can bias the output, since variations in the seed word list lead to differences in the final lexicon. To overcome this issue, we obtained the set of seed words in a data-driven way by estimating the frequency shifts (Gallagher et al., 2021) of the lemmas between the positive and negative documents, as done in Araque et al. (2022). This approach helps us to avoid the limitations of arbitrarily selecting the seed words. Thus, we consider the relative frequency of a word $w$ in a set of documents $D$:

$$p_{w}^{(D)}=\frac{f_{w}^{(D)}}{\sum_{w^{\prime}\in W^{(D)}}f_{w^{\prime}}^{(D)}}$$

(1)

where $w^{\prime}\in W^{(d)}$ are the words in vocabulary set $W^{(D)}$ except for $w$. We compute the frequency shift with relation to the relative frequency per word $w$ between two different sets of documents as:

$$\delta p_{w}=p_{w}^{(2)}-p_{w}^{(1)}$$

(2)

The seed word lists are generated based on prominent differences in word frequency shifts. We apply a minimum frequency threshold of 100 to filter out less common lemmas. We then use the word2vec algorithm (Le and Mikolov, 2014) to compute the vector for each word, using the standard parameter setting and a vector dimension of 300. The lexicon is generated by estimating the cosine similarity between the word vectors obtained using the emerging seed words. To compute the moral polarity of a word $w_{i}$ from the documents, we use the sets of seed words for the “oppressive” orientation ($S_{C}$) and the “liberty” direction ($S_{L}$), and estimate the polarity based on the cosine similarity:

$$\sum_{w_{j}\in S_{L}}\text{sim}(w_{i},w_{j})-\sum_{w_{k}\in S_{C}}\text{sim}(w_{i},w_{k})$$

(3)

where sim represents the cosine similarity as estimated by the word embedding model. The obtained polarity is positive if $w_{i}$ is related to the positive seed words and a negative value if the word is more related towards the negative seed words. For the rest of the paper, we refer to this model as the WE model and we generate one lexicon for each dataset (except Vaccine, which is only used for testing).

The second approach involves using the Compositional Semantics (CS) method (Liang et al., 2013), previously used to generate emotion lexicons Staiano and Guerini (2014); Araque et al. (2019). The CS method applies a projection of moral values from a document to its words. The underlying assumption is that each word is associated with the moral value present in the documents where the word appears more frequently.

To generate a word-by-moral association matrix ($M_{WM}$), we first create a document-by-moral matrix $M_{DM}$, which shows the distribution of the liberty foundation across the training dataset. We then generate a word-by-document matrix $M_{WD}$, which indicates the number of occurrences for each word in the vocabulary within a given document, normalized by the total number of words per document. To obtain the word-by-moral matrix, we perform a multiplication using the following expression:

$$M_{WM}=M_{WD}\cdot M_{DM}$$

(4)

Using this approach, words and their corresponding value of liberty can be merged by calculating the product of the weight of a word and the weight of the moral value in each document. The resulting scores are then normalised (column-wise), over-representation issues are addressed, and each lemma is scaled (row-wise) to sum up to one. Previous validation of lexicons has shown that this normalisation approach is suitable Araque et al. (2019, 2022). This approach is referred to as the CS model and we generate one lexicon for each dataset except Vaccine.

The domain-specific lexicons express the liberty dimension that is dependent on the topics of the dataset on which they are generated from. However, we are interested in deriving a general, higher-level representation, so that the final users of the resource have a unified and domain-independent resource. To this regard, we synthesise a unified resource merging the obtained discrete lexicons. This approach (i) augments the coverage of the consolidated lexicon and (ii) discards uncommon tokens and their annotations. The basic process to obtain such a lexicon starts by defining a unified vocabulary as the union of the vocabularies of all individual lexicons. This union can be controlled with a selection parameter expressed as a percentage value. That is, if we define a selection parameter of 50%, a word would be included in the union of vocabularies if it appears in at least the 50% of all considered lexicons.

Then, we align the numeric assignment each token has in the individual lexicon. We estimate the average score of these assignments, incorporating them into the unified resource, if the volume of annotations satisfies the threshold stipulated by the chosen proportion (selection parameter). We denote the obtained lexicon as Lexicon Overlap (avg.)⁵⁵5The lexicon will be released upon acceptance..

. We analyze the in-domain performance of our lexicons by testing them on a left-out set of the datasets they are generated from (consisting of 20% of the original data). Notice how, depending on the dataset, the task is slightly different due to the different type of annotations: (i) for the BLM and Elect. datasets, the classifier should predict whether the document expresses notions of liberty/oppression or is neutral; (ii) for the Reddit and WikiCon datasets, it should predict whether a document expresses the libertarian or the conservative point of view..

To avoid overfitting, each model is training on the training set of the respective balanced dataset (see Table 1), leaving the test set for evaluation.

. We perform a series of out-of-domain experiments, testing how well the lexicons can generalize to different domains. In particular, we measure the performance of: (i) the lexicons generated from Reddit and WikiCon used on the BLM and Election datasets; (ii) the lexicons generated from BLM and Elect. used on the Reddit and WikiCon datasets; (iii) the lexicons generated from BLM and Elect., trained on BLM/Elect. and tested on the Vaccine dataset.

Here the different annotation schemes and domains could potentially pose a bigger challenge for the classifier; however, while in the first two cases we can expect to see the impact of the different vocabulary, the train/test split is still coming from the same dataset (in other words, the model has to learn a task using sub-optimal features, but having “coherent” data for training and testing). The Vaccine dataset is instead used to test the out-of-domain performance of the lexicons when the annotation scheme of the evaluation dataset (i.e., the presence or absence of liberty foundation) is coherent with the annotation scheme of the dataset from which the lexicons are generated (BLM and Elect.), but no ideal training data is available (as we are training on an annotated dataset -BLM or Elect- that is different from the Vaccine test dataset).

For all experiments we utilized logistic regression (Alpaydin, 2020) and represented each document using a vector of the same length as each lexicon vocabulary. The feature vectors are constructed as follows; each document is represented by a vector of equal size to the lexicon. For those tokens in the document present in the lexicon, the vector contains the respective polarity score, otherwise zero. Since this type of representation dramatically simplifies the linguistic information present in the document, we enhance the classification design with two more experiments. We extend each vector representation with the “statistical summary” functions, namely the average, maximum, median, variance, and a peak-to-peak score of the lexicon values of that document. This offers the learning models a more complete view of the text.

. To test whether it is possible to obtain a more “general purpose model”, we evaluate two ways of combining the information coming from the different lexicons: (i) the lexicon overlap described in Section 4.2, which averages the values for words that appear in multiple lexicons; (ii) the combined representation, which is not a lexicon, but a method of learning a unified representation by taking into account all available lexicons.

The advantage of the first method is its simplicity, and that it results in an interpretable lexicon. On the other hand, the combined representation allows a learning model to observe simultaneously all information contained in the individual lexicons (including words not shared among them); the model might then be able to exploit existing interactions among them.

While the strength of this second approach is that it provides a comprehensive representation obtained through all individual lexicons, overfitting may occur given the large dimensionality of representation. To avoid such issue, we include a feature selection mechanism in the learning model so that the dimension of the feature vector can be reduced. Our approach is based on the Singular Value Decomposition technique (SVD) (Halko et al., 2011) for transforming the representation of a single lexicon into a continuous vector, which is then input to a machine learning algorithm, in our case a logistic regressor. Figure 1 illustrates the hierarchical structure of the proposed model.

Since these two approaches take into account all generated lexicons, we can consider the results of the lexicon overlap and combined representation methods: (i) an in-domain evaluation when applied to the BLM, Elect., Reddit or WikiCon datasets (since they are used to generate the individual lexicons), (ii) an out-of-domain evaluation, when applied to the Vaccine dataset (since this is only used as a test set).

As baseline models, we train two classifiers using a unigram representation that includes a frequency-selected vocabulary of sizes 1,000 and 10,000 tokens respectively. The two sizes are comparable to the size of the obtained lexicons (see Table 2).

To assess the general quality of the lexicons and obtain an overall ranking of their performance, we have performed the Friedman statistical test over all the evaluation results (Araque et al., 2017; Demšar, 2006). In the Friedman test a lower ranking implies a better result for a certain method in comparison to the rest. In case of ties, these are resolved by averaging the obtained ranks. The Friedman test has been performed with $\alpha=0.05$, rejecting the null hypothesis. We report the macro-averaged F-score as well as the Friedman rank for the overall evaluation of each resource.

Our expectation would be that the models trained on features emerged from lexicons generated on the respective data source would outperform the rest. However, we notice that this is not true in most cases, except when models trained with the Reddit and WikiCon train sets on features extracted from the Reddit and WikiCon lexicons respectively are employed to distinguish between notions of liberty or oppression.

Looking at these results, it can be seen that generally the learning models trained on the CS lexicon features improve over the unigram baselines, showing that these lexicons capture useful representations. In particular, CS lexicons are consistently on-par or above the baseline, when the lexicon and the dataset are coherent (e.g. lexicon features generated from the BLM dataset, trained and tested to predict the BLM dataset annotations).

In contrast, the overlap lexicon approach, combining information from different lexicons, shows a fairly consistent performance across all datasets. These results indicate that the difficulty of the task calls for combined knowledge, since enriching the representations with linguistic information from different contexts and writing styles improves the recognition of the liberty moral value in text.

We obtain further confirmation for this hypothesis by looking at the Friedman test: when considering all evaluation combinations, the Combined Representations ranks as the best approach, followed by the lexicon overlap. This is to be expected, as this method makes use of all lexicons simultaneously, and learns internal representations that can be exploited by a machine learning model.

At the level of individual lexicons, the Reddit one generated with the Compositional Semantics method achieves very good performance overall (being the third best performing lexicon in the Friedman rank) also when used on non-Reddit datasets. This may be an effect of the larger number of tokens and general quality of the original dataset, which probably includes a richer vocabulary for a variety of topics discussed by libertarians and conservatives. Besides, this observation offers the insight that, even though the annotations of the used lexicon are not completely aligned (e.g., using the Reddit lexicon for predicting Liberty/Oppression, while the Reddit dataset from which it is generated captures the Libertarian/Conservative divide), the knowledge captured by the lexicon can aid in the classification task. A possible explanation is that the lexicons cover the whole gamut of association strengths, thus capturing a balanced view of language and not just words strongly correlated with the liberty foundation; this could help the classifier learn the threshold between documents expressing this foundation (which will have more words with “extreme” values) and those which do not (probably consisting of mostly “neutral” words).

Regarding the “stricter” out-of-domain evaluation, the right side of Table 3 reports the results obtained when models fed with features extracted by the BLM, Elect, WikiCon, and Reddit lexicons (either CS or WE), are trained on the BLM and Elect. training dataset and tested on the Vaccine dataset. We notice that, again, the models trained on the Overlapping Lexicon are consistently performing well, while feature extraction based on individual lexicons led to models that did not consistently outperform the baseline.

This experiment offers interesting insights into the generalization capabilities of the proposed method. Although, the absolute best performance is obtained with the Elect. (WE) lexicon trained on the Election dataset, is not a generalisable finding; the same lexicon trained on the BLM data fails to outperform the baseline. Overall, this design offers insights on the adaptability and generalization capability of the lexicons. Interestingly, the combined representation approach ranks always high validating the fact that combining knowledge from different base lexicons does improve the understanding of the liberty foundation.

A recurrent pattern is that the overlapping lexicon outperforms the other lexicons in both in-domain and out-of-domain experiments. To gain more insights on the effect of the social context on the moral nuances a specific lemma may have, we employed the TOMEA approach proposed by Liscio et al. (2023). According to TOMEA, the overlapping lexicon differs the most with respect to the BLM lexicons generated by both the CS and the WE methods with scores .17 and .13, respectively. Glancing into the most distant words, we have “fake”, “lawmaker”, “elected”, “supporters”, “antifa”, “openly”, “tweets”, “sympathizer”, “tyranny”, “globalist”, “dead”, to be considered more oppressive than the average lexicons in the individual BLM lexicons than in the others. Such domain specific nuances may be important when analysing a specific argument but can introduce biased in the models when analysing broader subjects. TOMEA to this respect is a valuable method to gain insights and foster the transparency and accountability of the findings.