LLM-Collaboration on Automatic Science Journalism
for the General Audience

LLMs have shown strong writing abilities (Yuan et al., 2022; Wasi et al., 2024). Thus, they are promising tools for rewriting a provided paper into a more accessible version. Following established strategies (Zheng et al., 2024a; Zhang et al., 2024), we start with prompting an LLM to assume the role of a journalist. Subsequently, the LLM is prompted that, given the paper, its task is to compose an article for the general public.

In our preliminary attempts, we asked the journalist LLM to directly assess the readability of the content. However, the results are unsatisfactory, probably due to the gap between human and model perceptions of reading difficulty. As illustrated in the two text boxes at the bottom of Figure 3, LLMs may regard both pieces of writing are of a similar level of readability, as they all incorporate essential information, even if the terminology “low-cost paper-based microfluidic diagnostic tests” on the left side is not clearly explained. However, in the eyes of a human reader, the content on the right is perceived as more accessible.¹¹1We briefly document other failed attempts in Appendix B for reader information.

To address this readability assessment problem, we design a separate reader LLM to read the content and generate reading notes. Our idea is inspired by the accumulation of errors, a common phenomenon in pipeline systems (Caselli et al., 2015; Wu et al., 2018; Jiang et al., 2023b; Dziri et al., 2023). Specifically, we can utilize the propagation from the textual readability of the journalist’s article to the reading comprehensibility of the reader in the writing-reading pipeline, to induce the readability of the generated article to become explicit.

Different from the LLM journalist, the reader LLM is of a smaller scale and simulates a general reader with limited domain knowledge. Once presented with the article crafted by the journalist, the reader LLM is employed to read the article and take notes. Specifically, we instruct the reader LLM to explain key terms in the article by extracting the explanations, if present, directly from the article or offering explanations for these terms, otherwise.

Intuitively, if the article is more accessible, the reading notes will be more comprehensive. For instance, in Figure 3, if the piece (left) lacks a detailed explanation of the term “paper-based microfluidic diagnostic test”, the reader will only note this term as being “combined with deep learning”. If the article (right) explains the usage and advantages of this term in plain and readable language, the reader LLM can grasp this knowledge. Through this readability propagation from the journalist’s article to the comprehensiveness of the reader’s notes, the editor LLM can better recognize issues in the journalist’s writing and then provide suitable suggestions for modifications.

LLMs have demonstrated strong capabilities in serving as evaluators, widely utilized in various generative tasks (Chan et al., 2023; Zheng et al., 2024a; Desmond et al., 2024). Therefore, we employ an LLM as a senior editor for automated evaluation of reader comprehension and providing recommendations for article enhancement. Given the article from the journalist and notes from the reader, an LLM editor is tasked with assessing the quality of the reader’s notes and identifying issues in the journalist’s writing, which may lead to reading obstacles.

Next, the editor offers advice for the journalist’s content development. For example, in Figure 2, the editor finds that the reader’s understanding of the term “blockchain” is limited, possibly due to an insufficient explanation in the reading material. To address this perceived issue, the editor suggests that the article should “explain technical terms”. These suggestions are then incorporated into the instructions that will guide the journalist in revising the article.

Subsequently, with the strong ability to follow instructions, the journalist LLM rewrites the article according to the suggestions. Then, the revised piece is fed to the reader for reading and taking notes to continue the process. By an iterative cycle encompassing writing, note-taking, suggesting modifications, and revision among three LLMs, the article tailored for the general readership undergoes steady enhancement. We further analyze this process in Section 4.

We choose four representative methods for human evaluation: (1) plain summaries by human writers, (2) Qwen1.5-7B generation, (3) GPT-4 generation, and (4) generation by our LLM-Collaboration method. The human evaluation encompasses multiple dimensions, namely Readability, Information Conveyance, Authenticity, and Interestingness. Participants are tasked with evaluating the articles using a 1-5 Likert scale (Likert, 1932), grounded on specific questions. Each participant is assigned to assess all articles both in the field they are familiar with and those they are not familiar with, to provide a genuine evaluation from readers within the specific discipline and general readers. Appendix D shows the details of these measures and the questionnaire form.

The results in Table 1 show the comparison of different methods. Recent LLMs of similar scales have shown comparable performance, surpassing the LLaMA-2 introduced in 2023. Larger LLMs such as Mixstral-8x7B and Qwen1.5-72B show even better performance, indicating that performance improves as the model scale increases. Additionally, the formidable LLM GPT-4 outperforms all the other single LLMs. These findings demonstrate their performance on ASJ consistent with the capacity of LLMs. The fine-tuning methods exhibit competitive performance, slightly superior to prompting the open-source LLMs. Fine-tuning a model with a larger scale (Qwen1.5-7B) outperforms fine-tuning BART, in line with prompting.

Our collaboration with LLMs has demonstrated significant improvement over previous methods. Interestingly, it can be observed that fine-tuning the journalist LLM to warm up does not lead to any noticeable improvement. One possible reason may be the decay in the ability to follow instructions after this specialized training. Nonetheless, these empirical results show the effectiveness of our framework that integrates multiple LLMs.

For a thorough evaluation, we carry out a human assessment on four representative methods. We request that participants assess articles relevant to their fields as well as those in their unfamiliar fields. The results are reported in Table 2, categorized into within-field and outside-field articles. Notably, participants assign lower ratings to articles outside their expertise, possibly due to the inherent comfort and familiarity bias elevating subjective scores. Despite the evident discrepancies arising from differing reader familiarity contexts, the method comparisons and overarching trends appear consistent across both settings.

Interestingly, all LLM-based methods outperform the plain summaries written by humans, probably because these summaries remain technical, providing a poor reading experience for a wide readership. The LLM-generated content, targeted to popular science, should be easier for both in-domain and out-of-domain readers to read. Furthermore, our LLM-collaboration approach surpasses the single Qwen in all dimensions, demonstrating the effectiveness of LLM collaboration. Most importantly, our LLM collaboration method achieves the highest readability and interest in both within-field and outside-field reading evaluation, with high information conveyance and authenticity close to the most advanced model GPT-4. Collectively, these findings attest to the potency and effectiveness of our proposed approach.

We conduct an ablation experiment to validate the effectiveness of each component. The results have been included in Table 6 for clarity. In the “Reader: 1.8B $\rightarrow$ 7B” setting, we substitute the 1.8B reader model with the 7B version. This substitution leads to a minor performance fluctuation. On the one hand, the 7B model’s strong ability gives it greater tolerance for low readability of content, making it harder to highlight writing issues in articles. On the other hand, it excels in instruction following, enhancing task execution and reducing intermediate errors in the workflow. The dynamics between gains and losses render the use of the 7B model as a “reader” comparably advantageous at times and disadvantageous at others. Nevertheless, we recommend using the 1.8B model due to its higher resource efficiency and throughput.

For $-$Reading Notes, we eliminate the requirement for the reader LLM to read the article and make notes. Instead, we ask the editor LLM to offer suggestions directly. For $-$Suggestions, the editor’s advice is omitted, and the journalist revises the article based on the reader LLM’s reading. For $-$Collaboration, the journalist revises the article based on the previous writing without any input from the reader or the editor. As depicted in Table 6, our approach exhibits a decrease in performance when each module is removed, underscoring the significance of each module.

We provide an in-depth analysis of the performance trajectory over iteration cycles. The entire writing-reading-suggesting-revising process is carried out for five iterations, with readability score changes depicted in Figure 4. Our findings reveal a successive and pronounced decline in all reading difficulty metrics over the initial three iterations. This pattern demonstrates the efficacy of our iterative revision methodology. Following the third iteration, the performance levels off, indicating diminishing improvements from subsequent suggesting and editing efforts.

To facilitate an intuitive comparison of our method, we present a case study on one writing sample. As shown in Table 3, our method can generate articles that are more readable, with concise and vivid expressions, along with explanations for technical details. For instance, our generated article states that “timely diagnosis is crucial” rather than “results need to be communicated rapidly to healthcare professionals once testing has been completed so that care pathways can be implemented”, making it more brief and accessible for readers to grasp the research objective. Moreover, our generated article details that the proposed system is a “smartphone-based system that combines a low-cost test with deep learning and blockchain technology” rather than “smartphone-based end-to-end platform for multiplexed DNA diagnosis of malaria”, enhancing the comprehensibility for a broader audience.

We also present a case study on the notes taken by the reader LLM during the first three iterations. As evidenced in Table 4, the notes offer more detailed explanations for technical jargon than those in preceding iterations. For example, the technical terms “AI” and “blockchain” are more thoroughly explained in the third iteration than in the first two iterations. This phenomenon suggests that as the readability of writing improves, readers can acquire deeper and more elaborate insights, aligning with human reading behaviours.

We further examine the advice given by the LLM editor for the revision over the iterative process, as exemplified in Table 5. In the initial stage, the editor offers general advice. Following the revision, the advice becomes more specific and detailed. Subsequent rounds of suggestions highlight specific content in the article and recommend revisions based on various aspects. In the 3rd iteration, the feedback from the editor becomes more detailed and specific, with specific adjustments from the original version to the revised content. This revision behaviour is similar to real-world science journalism, showcasing the efficacy of our framework and the social intelligence of LLMs.