BiasScanner: Automatic Detection and Classification of News Bias to Strengthen Democracy^††thanks: The authors gratefully acknowledge the funding provided by the Free State of Bavaria under its “Hitech Agenda Bavaria”. All views are the authors’ and do not necessarily reflect the views of any funders or affiliated institutions. We thank Michael Reiche for feedback and discussions.

Democracy faces an existential threat when most citizens get their news from online platforms focused on controversy rather than balanced reporting. Such behavior increases advertising revenue, contributing to media bias and the spread of fake news [20, 15, 25, 14, 31].

To combat this trend, we introduced BiasScanner, a practical tool to help readers assess online news regarding instances of biased reporting, which we describe here. It highlights biased sentences, offers detailed analysis reports, and assigns bias scores. Users can also donate bias reports for research. BiasScanner makes use of advanced neural transformer models, such as OpenAI’s GPT 3.5 for efficient and effective bias detection. We prioritize user privacy by not storing personal(ly identifiable) information or news stories without explicit consent.

Foundational Language Models. The neural transformer model was first described in [30]. Google BERT [11] and OpenAI’s GPT-3 [5] GPT-4 (and their application ChatGPT [24]) and Meta’s Llama [29] have been early foundational models that have introduced a paradigm shift in NLP by demonstrating how large, pre-trained language models can dramatically reduce the development time of NLP systems by using large quantities of un-annotated text to train general-purpose “foundational” models.

News Bias. Groeling [14] presents a survey of the literature covering partisan bias. Conrad et al. [8] focused on content mining to measure credibility of authors on the web. The topic of bias in mass media was dealt with in detail by [20] and [25]. Hamborg et al. [16] provided an interdisciplinary literature review to suggest methods how bias could be bias detection could be automated.
Bias Detection. Media bias datasets with different focus where released by [2],[17], and [27, 26] After early pioneering work on bias from economics [15], Arapakis et al. [2] labeled 561 articles along 14 quality dimensions including subjectivity. Horne et al. [17] released a larger dataset annotated for political partisanship bias, but without grouping articles by event, which makes apples-to-apples comparison harder; Chen et al. [6] addressed this issue by resorting to another corpus sampled from the website allsides.com, which includes human labels by U.S. political orientation (on the ordinal scale $\{LL,L,C,R,RR\}$); they also present an ML model to flip the orientation to the oppositite one. Yano, Resnik and Smith [33] also on the liberal-conservative axis, manually annotating sentence-level partisanship bias.
MBIB, the first media bias identification benchmark, was introduced by Wessel et al. [32], who evaluated Transformer techniques on detecting nine different types of bias across 22 selected datasets. Baumer et al. focused on detecting framing language. [3] Chen et al. [7] demonstrated that incorporating second-order information, such as the probability distributions of the frequency, positions, and sequential order of sentence-level bias, can enhance the effectiveness of article-level bias detection, especially in cases where relying solely on individual words or sentences is insufficient. Spinde et al. published a dataset containing biased sentences and evaluated detection techniques on it.
[27, 26]

Web Apps & Mobile Apps. Hamborg et al. [16] presented Newsanalyze, a system that highlights sentiment target entities colored by polarity. In contrast, we perform sentence classification targeting bias and sub-type of bias (sentiment $=$ affective state $\neq$ bias (Although there can and typically is a connection, bias is more general e.g. under-reporting is not sentiment-related at all). Da San Martino et al [10] developed Prta, a tool highlighting propaganda techniques in news articles. While propaganda and news bias are related (as visible in the overlap of propaganda techniques and bias types), new bias is a broader phenomena, also including unintentional subjective reporting.
Other Related Work. Conrad, Leidner and Schilder characterize signals for credibility in the context of credibility for professionals [9]. Bhuiyan et al. [4] compare crowdsourced and expert assessment criteria for credibility on statements about climate change. Allen and co-workers [1] studied the Ghanem et al. [13] analyze an interesting way to distinguish between real/credible news and fake news by looking at the distribution of affective words within the document.

To the best of our knowledge, BiasScanner is the first system for news bias detection and bias sub-type classification based on a neural transformer architecture published in the scientific literature and deployed/release to the general public as a free browser plug-in.

This section describes BiasScanner, our system, which is also deployed on the World Wide Web at https://biasscanner.org. This address also contains a separate Web demo where users can experiment with our model before installing the Web browser plug-in.

Quantiative Evaluation. While a detailed evaluation is beyond the scope of this system paper, we presented detailed quantitative and qualitative evaluations for the English language in [21] and [22]. Table 1 shows some quality numbers from [22], which were representative as of June 2024 (for BiasScanner as of release 1.0.0 from July 2024; ongoing development may lead to different scores going forward.) F1-score is high at 76%, and our fine-tuned model’s quality dominates GPT-4 on all metrics except for precision (73% versus the latter’s 85%, at the time of writing).

Qualitative Evaluation. The achieved quality level is satisfying for practical use of the browser plug-in; a common error is the mis-classification of neutral reporting sentences with embedded radical quotes as “biased”; we believe embedded quotes ought to be removed before judging a sentence, which we will address in future work. We are particularly encouraged by the quality of our generated explanations, the evaluation of which is left for future work.

Beyond English. At the time of writing, BiasScanner can deal with news in English through our fine-tuned model, and also with other languages via said model’s transfer capabilities; in future work we want to fine-tune models for additional specific languages and evaluate them, as well as compare their performance with our existing model’s transfer abilities.

BiasScanner may not identify all instance of biases, and while we do not claim it does, the users may wrongly believe otherwise, consciously or unconsciously, after getting used to it. It can also not recognize all types of bias: notably, underreporting bias and other types that need across across several articles, are beyond its scope, as it only analyzes one individual news story at a time; we leave news coverage comparison for future work. It should also be noted that bias detection is always, to an extent, a subjective matter. Often a sentence might be considered biased by one person while another considers it to still be objective, therefore no classification will probably ever satisfy everyone at once.

Our current back-end implementation still depends on an underlying proprietary foundational model; in future work, we plan to become independent and port to an open model, even if this may mean a slight reduction of accuracy, as this may limit the ability to manipulate the system’s behavior from the outside.

We introduced BiasScanner, a new system for enhancing online news consumption by highlighting biased individual sentences in news articles, by offering news story analysis within Web browsers. We have successfully realized our design goals, including user privacy, rapid implementation and accurate bias classification.

BiasScanner may not identify all biases, as to date it focuses on individual news stories and does not compare across articles.

To date, BiasScanner has mainly been tested with English articles, introducing a development bias. Sending plain text to a server for security is required, but it is done anonymously. The system has been released as experimental browser extension available free of charge for Firefox trough the Mozilla plug-in marketplace[18] (Available on Desktop and Android). Future Releases for Chrome and Safari are planned. It can also be installed from GitHub [19].

We are also already using BiasScanner in the classroom for the teaching of critical reading and engaging students with the topic of media manipulation and its effects on a democracy (in Summer Semester 2024, the second author used it to support his course Media Manipulation, Propaganda and Fake News at Coburg University of Applied Sciences in Germany).

In future work, we aim to support open-source language models [29] to reduce cost and decrease reliance on commercial model vendors. We intent to support languages other than English, and we plan to expand the tool’s capabilities for multi-dimensional content analysis, including hate speech detection, readability scoring, fake news detection/credibility assessment and identifying inappropriate content for children [12]. We also welcome collaborations with other research teams and contributions to our effort from the open source community.