BioASQ is "a competition on large-scale biomedical semantic indexing and question answering"[4]. It is held as a lab at the Conference and Labs of the Evaluation Forum (CLEF) conference⁸⁸8https://clef2024.clef-initiative.eu/. The current 2024 workshop is the 12th installment of the BioASQ competition⁹⁹9http://www.bioasq.org/.

The 12th BioASQ challenge comprises several tasks:

• •

  BioASQ Task Synergy On Biomedical Semantic QA For Developing Issues [5]

• •

  BioASQ Task B On Biomedical Semantic QA [5]

• •

  ioASQ Task MultiCardioNER On Mutiple Clinical Entity Detection In Multilingual Medical Content [6]

• •

  BioASQ Task BioNNE On Nested NER In Russian And English [7]

We participated in Task B and Synergy[5]. For Task B the participants’ systems receive a list of biomedical questions that should be answered with a short paragraph style answer and some require an additional exact answer which can be one of 3 formats, yes/no, factoid (a list of up to 5 entities) or list (a list of up to 200 entities). Additionally, the systems first have to retrieve relevant papers from the PubMed annual baseline and extract relevant snippets from these papers that could aid in answering the questions. This retrieval subtask of Task B is called Phase A while the actual question answering subtask is called Phase B. For Phase B the systems also receive a set of gold snippets and documents that should help them answer the question. Task B was scheduled in 4 batches with two weeks in between and ran from March 28 to May 11.

In the 12th installment, another Phase A+ was introduced, where the systems were supposed to provide answers to the questions before the gold snippets and documents were provided, relying solely on their own retrieved documents and snippets.

For the Synergy task, the systems receive a similar list of questions, for which they also have to retrieve useful papers and extract snippets and as soon as a question is marked as ready to answer, they also need to submit answers in the same format as for Task B. The difference between Synergy and Task B is that initially in the first round no gold set of documents and snippets is provided, instead the submitted documents and snippets by the systems are evaluated by biomedical experts and selected as gold reference items for subsequent rounds. This also means that the same questions might be reintroduced in subsequent rounds, possibly with additional questions and positive and negative feedback on the previously submitted documents.

Following this introduction, we will highlight some related work in Section 2, describe our methodology in Section 3, report our results in Section 4 and discuss them in Section 5. Section 6 will present some ethical considerations, and Section 7 offers our conclusions.

Nearly all the popular SOTA LLMs that are used across various NLP tasks and use cases today are based on the transformer architecture [8]. With the generative pretrained transformer (GPT) [9] being a popular variant. These models undergo pre-training on vast amounts of text by solving the next-token prediction task [9]. Afterwards, the models are fine-tuned to align with human preference data [10] which enables them to follow instructions and be useful in direct interactions with users.

OpenAI was the first company that released such a fine-tuned model to the public in November 2022¹⁰¹⁰10https://web.archive.org/web/20240502090536/https://openai.com/index/chatgpt/ which sparked massive interest in generative artificial intelligence research and products. Their latest model at the time of writing that is powering their ChatGPT product was GPT-4 [11]. One interesting competitor model that we also used during this competition is Claude 3 Opus¹¹¹¹11https://web.archive.org/web/20240516173322/https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf by Anthropic, which reached GPT-4 level performance (GPT-4-0125-preview) at the time of the BioASQ competition¹²¹²12https://chat.lmsys.org/?leaderboard. The exact architecture of GPT-4 and Claude Opus 3 and other commercial models is unknown.

GPT-4 is the most expensive and slowest model that OpenAI is offering via their API service. A more affordable alternative that they are offering is GPT-3.5-turbo. We compared both these models’ performance in last year’s BioASQ competition and were able to show that GPT-3.5-turbo was sometimes performing better than GPT-4 in some question formats and subtasks of the competition [12].

For this year’s BioASQ competition we also used Mixtral 8x7B [13] a downloadable open-source model (Apache 2.0 license) which uses a type of Mixture-of-Experts Architecture [14][15]. This architecture offers higher computational efficiency by routing requests to expert subnetworks to generate a response. Since only some specialized experts are active during the generation, less computation and memory is needed to serve the request.

Few-Shot learning is the ability of LLMs to learn how to solve a new problem that they were not specifically fine-tuned for by only showing them a few examples. When GPT-3 was first introduced, its impressive few-shot learning abilities made the concept popular [16] because it greatly reduces the need for expensive training data.

Zero-shot learning [17] takes this concept a step further by only requiring an abstract task description or direct question, which leads the model to ideally generate a useful completion that solves the task at hand [18]. In last year’s BioASQ competition, we were able to win some batches while only using zero-shot learning with SOTA commercial models [12].

Current LLMs have billions of parameters and require specialized hardware with enough GPUs and VRAM to hold all the model weights in memory. For example, "16-bit finetuning of a LLaMA 65B parameter model requires more than 780 GB of GPU memory"[19]. Since this makes fine-tuning these models prohibitively expensive for many researchers and users, several clever techniques have been invented to reduce these hardware requirements. We wanted to fine-tune Mixtral 8x7B, which roughly takes up the memory of a 47B model¹³¹³13https://mistral.ai/news/mixtral-of-experts/.

One popular approach is QLora by Dettmers et al. [19] where the model weights are quantized to 4 bits and frozen and only some low rank adapters (LoRa)[20] are fine-tuned. This would enable fine-tuning Mixtral 8x7B, for example, on only two RTX A6000 GPUs with 2x 48 GB of VRAM.

Retrieval augmented generation (RAG) is a technique [2] that combines information retrieval with language models to enhance their ability to generate relevant and factual text. In RAG, the language model is augmented with an external knowledge base or other information source, such as a collection of documents or web pages. When generating text, the model first retrieves relevant information, based on the input query, and then uses that information to guide the generation process. This process is applied in the BioASQ challenge, where the relevant information source is the annual baseline of PubMed.

RAG has been shown to improve the factual accuracy of generated text compared to standalone language models [21]. It allows the model to access a vast amount of external knowledge and incorporate it into the generated output. RAG is particularly useful for tasks that require domain-specific knowledge or up-to-date information [3].

Professional search is conducted in a professional context, often to aid in work-related research tasks [22]. In some professional search settings, highly trained specialists are needed to create documented and reproducible search strategies, this sets professional search apart from everyday web search [23]. The BioASQ challenge exemplifies one possible professional search setting where biomedical experts aim to find answers to domain-specific questions with sufficient evidence.

Other examples of professional search might be systematic reviews [24], patent-search or search conducted by recruitment professionals [25]. All of these settings might require complex search strategies, where the search expert makes use of a query syntax involving boolean operators on specific search fields. Systematic reviews, for example, also require the search to be explainable and reproducible, which makes it difficult to use advanced vector-based retrieval techniques. Formulating traditional queries but with large language models that might be able to expand synonyms and related terms based on their semantic representations is therefore an interesting approach that might aid in professional search settings. We set out to explore this approach in the BioASQ challenge.

In this year’s BioASQ competition, we looked at the commercial offerings GTP-3.5-turbo and GPT-4 from OpenAI and also used Antrophics Claude 3 Opus, which was at the time of the run submissions the only other model that was on a level with GPT-4 according to the LMSYS Chatbot Arena Leaderboard [26].

Since last year’s BioASQ competition, some competitive Open-Source models were published. The most notable ones being the Llama series models by Meta [27], with the latest being Llama 3 [28]. Llama comes with its own custom License which is quite permissive, but even though commercial use is allowed under this license, as long as the monthly user base does not exceed 700 million users, the license might not be straightforward to adopt for enterprise use cases when licenses have to be pre-approved by a legal team.

When we prepared our runs for the competition, the best-performing model with a permissive open-source license (Apache 2.0) on the LMSYS leaderboard was Mixtral 8x7B [13]. The model also has a large context length of 32k tokens, which makes it especially interesting for RAG use cases or few-shot learning. We therefore choose Mixtral 8x7B as our open-source competitor model for this competition. During the competition, the newer Mixtral 8x22B model was also published, and we used it in some batches of Task B.

We used the commercial hosting service fireworks.ai¹⁴¹⁴14https://fireworks.ai/ to access and fine-tune Mixtral 8x7B, as the provided speed was very high and costs for their API usage were low.

We downloaded and indexed the annual PubMed baseline from the official website¹⁵¹⁵15https://pubmed.ncbi.nlm.nih.gov/download/. We indexed both the title and the abstract of all papers in separate fields of our index using the built-in English analyzer of Elasticsearch¹⁶¹⁶16https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html#english-analyzer. For every round of synergy, the most recent snapshots of 2024 up to the date considered in that round were downloaded and indexed in another similar index, which was then also searched during the runs.

For synergy, we used both gpt-4-0125-preview and gpt-3.5-turbo-0125, the newest available versions of OpenAI’s GPT-4 and GPT-3.5-turbo at the time of the competition. We used 2-shot learning to generate queries for our Elasticsearch PubMed index and zero-shot learning for extracting and reranking snippets as well as answering questions. We also wanted to use Mixtral 8x7B in this task, but the model was unable to follow instructions well enough to produce usable runs, especially in the zero-shot setting.

Given a question, we prepended the prompt in Listing 1 with two examples where the same prompt contained other questions, for example "Is CircRNA produced by back splicing of exon, intron or both, forming exon or intron circRNA?" and an ideal completion in form of an Elasticsearch query for the query_string endpoint of Elasticsearch for which an example can be seen in Listing 2. We sent both the examples and the prompt with our actual question to the model and received back a JSON object that could be used to directly query our index.

We ran the generated query to retrieve the top 50 relevant documents from Elasticsearch. We filtered out documents that were marked as irrelevant in the feedback file for the synergy round. We then sent each remaining article alongside the question to the model and used a zero-shot prompt to extract a list of relevant snippets from the article. We then used string matching to insure the returned snippets were actually present in the article title or abstract and to calculate the offsets.

We collected all relevant snippets from all potentially 50 articles and filtered out articles as irrelevant where the model did not extract any snippets from. We then prompted the model to select the top 10 snippets ranked by helpfulness from the set of snippets. We finally reranked the retrieved articles according to the snippet order returned by this step.

In the question answering step, we used the identical zero-shot prompts from our last year’s participation in Task B for this year’s Synergy task [12]. And we also merged the already deemed relevant snippets from the feedback files into our list of snippets that we passed on to the modal alongside the question prompt.

We also sent the same initial system prompt that we used last year [12] to the models. For the parameters, we set the temperature parameter to 0 to reduce randomness in the completion and supplied a seed parameter which is a new feature offered by OpenAI that can help maximize reproducibility for the model output, but determinism is still not guaranteed¹⁷¹⁷17https://platform.openai.com/docs/api-reference/chat/create#chat-create-seed. We also used the new response_format parameter to insure the model produced valid JSON responses for the prompts where we needed it¹⁸¹⁸18https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format. The exact python notebooks with all the implementation details and prompts used are available in our GitHub repository.

For Task B we reused the indexed PubMed annual snapshot that we created from the synergy task. We also switched the models, we added Mixtral 8x7B Instruct v0.1 as an open-source model, and we used Claude 3 Opus instead of GPT-4 because it became available shortly before the Phase started, and we had access to a free beta evaluation account.

In batches 1 & 2 we also explored the fine-tuning service of OpenAI and created 6 fine-tuned versions of GPT-3.5-turbo for each sub problem that our system had to solve. We also created 6 QLoRa fine-tuned versions of Mixtral 8x7B Instruct v0.1 using the fine-tuning service of fireworks.ai. The training sets were created from the supplied training data. We also adjusted our code to be able to use these training files as sources for few-shot examples.

The sub problems that we sampled training sets for were:

• •

  Snippet Extraction

• •

  Snippet Reranking & Selection

• •

  Summary Question Answering

• •

  Exact yes/no Question Answering

• •

  Exact factoid Question Answering

• •

  Exact list Question Answering

In batches 3&4 we explored how we could use the models to retrieve relevant additional context from Wikipedia. We hypothesized that supplying these models with knowledge about relevant entities in the questions might improve their ability to generate correct answers. We wanted to explore the approach of retrieving such additional information about entities from a Wiki, because such a wiki like could also be generated in an enterprise setting, potentially closing the knowledge gap for entities that the models didn’t encounter during pre-training. We chose Wikipedia as a knowledge base even though the concepts described there might not be novel to the models because it was easy to use, and we hoped that we could observe an effect even with known concepts. We suspected that these models might just "know" which Wikipedia articles might be relevant to a question because they might have been highly trained on Wikipedia and links to Wikipedia articles. The exact zero-shot prompt that was used for finding relevant Wikipedia articles can be seen in Listing 3.

The titles of the Wikipedia articles returned by this prompt were extracted and, if the Wikipedia articles actually existed, their content was retrieved and concatenated. Finally, the concatenated articles were again sent to the model, and it was prompted to make a concise summary to help answer the question. This summary was then added as additional context to all subsequent prompts such as query generation or snippet extraction, reranking and question answering.

We participated with 2 systems in batch 3 and 4 of the Synergy task. The full result table is accessible on the BioASQ website²⁰²⁰20http://participants-area.bioasq.org/results/synergy_v2024/. The two systems both used the same 2-shot query expansion and zero-shot Q&A approach but with different commercial models. The system names and corresponding models are listed below:

• •

  UR-IW-2: gpt-4-0125-preview.

• •

  UR-IW-3: gpt-3.5-turbo-0125

We tried to also use Mixtral 8x7B Instruct v0.1 as an open-source alternative in this task, but the model was unable to follow the zero-shot prompts for query expansion and extracting the snippets consistently enough to produce a submittable run file. Our retrieval approach for query expansion and filtering and reranking via extracted snippets was not competitive in the document retrieval stage of the task, with both models performing similarly poorly compared to the top competitors, as indicated in Table 1. The official metrics to rank systems in each subtask are highlighted in bold in the following tables²¹²¹21”Top Competitor” are the systems that took the first position in a round or batch that are not ours. They are added as a reference point for the reported metrics. When ”Top Competitor” is missing in a reported batch, one of our systems was the best-performing one..

For snippet extraction, the performance of our approach was also poor, except for batch 4 where gpt-3.5-turbo-0125 was able to achieve second place as can be seen in Table 2. Gpt-4-0125-preview was unable to extract any snippets in the same batch, which was unusual and could have been due to issues OpenAI had with serving the preview model via their API.

For the question-answering stage, both models achieved perfect scores in the exact yes/no answer format, see Table 3. For factoid answers, gpt-4-0125-preview was able to take first place in batch 4 and achieved higher placements than gpt-3.5-turbo-0125 over both batches, as can be seen in Table 4. A similar difference is also observable in Table 5 for the list answer results.

Completing one run with gpt-4-0125-preview cost around $12 in API fees, while the same run with gpt-3.5-turbo-0125 was around 10 times cheaper at $1.2. Gpt-4-0125-preview was also quite slow, taking around 180 minutes to complete one run, while gpt-3.5-turbo-0125 took only a few minutes. Cost significantly decreased compared to our last year’s participation, while speed increased. This enabled us to actually use GPT-4 on the snippet extraction task, while last year we were not able to complete runs with snippet extraction for GPT-4 due to time and cost constraints. We also encountered less to none API errors during our runs, except for the empty responses in batch 4 for GPT-4, while last year we often had to rerun questions because the API timed-out or returned other errors.

We participated with 5 systems in all 4 batches of Task 12 B Phase A. The systems either used 1- or 10-shot learning with the plain or a fine-tuned model in batches 1+2 or additional context retrieved from Wikipedia in batches 3-4. The system names and configurations are listed below.

Batches 1-2:

• •

  UR-IW-1: Claude 3 Opus + 1-shot

• •

  UR-IW-2: Mixtral 8x7B Instruct v0.1, QloRa Fine-Tuned + 10-Shot

• •

  UR-IW-3: gpt-3.5-turbo-0125 fine-tuned + 1-shot

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 + 10-shot

• •

  UR-IW-5: gpt-3.5-turbo-0125 + 10-shot

Batches 3-4:

• •

  UR-IW-1: Claude 3 Opus 1-shot + wiki

• •

  UR-IW-2: Claude 3 Opus 1-shot

• •

  UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot

• •

  UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki

The following Tables 6 and 7 show the results of our systems participating in the 4 batches. MAP was the official metric to compare the systems.

In batches 1 & 2 where we compared fine-tuned versions of GTP-3.5 and Mixtral 8x7b with 10-shot learning and Claude 3 Opus, no clear trend was observable over batches in the document retrieval stage of Phase A as can be seen in Table 6. Only Mixtral 8x7B with 10-shot learning was consistently performing worse than all our other models, While Claude 3 Opus with 1-shot learning was our best model in batch 2 and second best in batch 1.

We used 1-shot learning instead of 10-shot learning for Claude 3 Opus due to time constraints because the model was slow, and for the fine-tuned gpt-3.5-turbo due to cost constraints. Sending 10 examples of abstracts for snippet extractions per 50 highest-ranked search results would have amounted to quite some input tokens per run for these models.

For batch 3 & 4 where we explored if giving the systems additional Wikipedia context while creating queries for Elasticsearch could improve their performance, we also could not observe a consistent effect over batches. While in batch 3 the systems with additional Wikipedia context performed better, this effect was reversed in batch 4.

In the snippet extraction stage of Phase A, our QloRa fine-tuned version of Mixtral 8x7B was our worst-performing system in both batch 1 & 2, followed by the 10-shot Mixtral 8x7B version as can be seen in Table 7. The fine-tuned version of GPT-3.5-turbo was consistently ahead of its 10-shot counterpart in both batches, while Claude 3 Opus was worse than the GPT-3.4-turbo systems in batch 1 and better in batch 2.

Additional Wikipedia context did not lead to consistent results across batches. While the systems with Wikipedia context (UR-IW-1, UR-IW-3) performed better than their counterparts (UR-IW-2, UR-IW-4) in batch 3, this effect was again reversed in batch 4.

One run with Claude 3 Opus with 1-shot learning and additional Wikipedia context took around 140 minutes to complete, we did not have to pay for the tokens used because we had an early beta evaluation account. For Mixtral 8x7B with 10-shot learning and additional Wikipedia context, the runs took around 14 minutes to complete via the fireworks.ai API and cost around $ 11. Fireworks.ai charged $0.50 /1M tokens for both input and output tokens as of writing, while Anthropic would have charged $ 15 /1M tokens for input and $ 75 /1M tokens for output. So the cost for doing 10-shot learning with Claude 3 Opus would have been at least 30 times as high while being 10 times slower.

We participated with 5 systems in nearly all 4 batches of Task 12 B Phase A+²²²²22We failed to submit one system run for system number 5 in batch 3.. The systems either used 10-shot learning with the plain or a fine-tuned model in batches 1+2 or additional context retrieved from Wikipedia in batches 3-4. Per batch, we used the same input snippet file for all systems to base their answers on to ensure that their performance is comparable.

Batches 1-2:

• •

  UR-IW-1: Claude 3 Opus + 10-shot

• •

  UR-IW-2: Mixtral 8x7B Instruct v0.1, QLoRa Fine-Tuned + 10-Shot

• •

  UR-IW-3: gpt-3.5-turbo-0125 fine-tuned + 10-shot

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 + 10-shot

• •

  UR-IW-5: gpt-3.5-turbo-0125 + 10-shot

Batches 3-4:

• •

  UR-IW-1: Claude 3 Opus 10-shot + wiki

• •

  UR-IW-2: Mixtral 8x22B Instruct v0.1 10-Shot

• •

  UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot

• •

  UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki

For the yes/no exact answer format, Claude 3 Opus with 10-shot learning was our worst-performing system, while the fine-tuned version of GPT-3.5-turbo was our top-performing system in batch 1 and only beaten by its 10-shot counter-part in batch 2, as can be seen in Table 8. It was interesting to see that the open-source models could perform better than the presumably most advanced commercial model, Claude 3 Opus, in this task.

For batches 3 & 4, we could show that additional Wikipedia context led to inconsistent results. While this context improved performance in batch 3 for the Wikipedia enhanced systems (UR-IW-5, UR-IW-3) over their normal 10-shot counterparts (UR-IW-2, UR-IW-4) it again led to worse performance in batch 4. This result is in line with the results from Phase A where these systems performed similarly for document retrieval and snippet extraction. We speculate that the models are sensitive to the Wikipedia context, and the usefulness of the context is highly influenced by both the entities present in the questions, and its relationship to the relevant snippets.

In the exact answer factoid format, our worst-performing system in batches 1 & 2 was consistently Mixtral 8x7B with 10-shot learning while its fine-tuned counterpart performed better as can be seen in Table 9. This order was reversed for GPT-3.5-turbo, where the fine-tuned version performed worse than its counterpart.

The additional Wikipedia context again led to inconsistent results across batches, but this time the systems with Wikipedia context performed better in batch 4 compared to batch 3, which is contrary to the observed behavior in the document retrieval and snippet extraction in Phase A as well as the yes/no answer format in Phase A+.

For the list exact answer format, Mixtral 8x7B was again our worst-performing system in batch 1 & 2, while its fine-tuned counterpart was competing with the fine-tuned version of gpt-3.5-turbo for the top positions as can be seen in Table 10.

In batches 3 & 4 Claude 3 Opus with 10-shot learning and additional Wikipedia context was the best-performing system in both batches while Mixtral 8x7B with 10-shot learning and additional Wikipedia context was the worst-performing one. Overall no consistent effect of the Wikipedia context was observable across models and batches.

We participated with 5 systems in all 4 batches of Task 12B Phase B. The systems used 10-shot learning with the plain or a fine-tuned model in batches 1 & 2 or additional context retrieved from Wikipedia in batches 3 & 4.

Batches 1-2:

• •

  UR-IW-1: Claude 3 Opus + 10-shot

• •

  UR-IW-2: Mixtral 8x7B Instruct v0.1, QLoRa Fine-Tuned + 10-Shot

• •

  UR-IW-3: gpt-3.5-turbo-0125 fine-tuned + 10-shot

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 + 10-shot

• •

  UR-IW-5: gpt-3.5-turbo-0125 + 10-shot

Batch 3

• •

  UR-IW-1: Claude 3 Opus 10-shot + wiki

• •

  UR-IW-2: Mixtral 8x22B Instruct v0.1 10-Shot

• •

  UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot

• •

  UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki

Batch 4

• •

  UR-IW-1: Claude 3 Opus 10-shot + wiki

• •

  UR-IW-2: Claude 3 Opus 10-shot

• •

  UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki

• •

  UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot

• •

  UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki

In the exact yes/no answer settings of Phase B, Claude 3 Opus with 10-shot learning and gpt-3.5 turbo with 10-shot learning were sharing first place in batch 1 while the fine-tuned version of gpt-3.5-turbo was the best-performing system in batch 2 as can be seen in Table 11. The fine-tuned version of Mixtral 8x7B was also competitive, taking the 5th position in batch 1 and 8th position in batch 2.

In batch 3 & 4 with additional wikipedia context the systems performed better than their counterparts in batch 3 while this result was mixed in batch 4, leading again to inconsistent results.

For the exact factoid answer format in Phase B, Claude 3 Opus and our fine-tuned Mixtral 8x7B model where sharing first place in batch 2 while in batch 3, Claude 3 Opus and gpt-3.5-turbo where the on the same level as can be seen in Table 12.

For Mixtral 8x7B, additional Wikipedia context improved the outcome in batch 3 but led to worse results in batch 4, while a similar effect was observable for Claude 3 opus in batch 4²³²³23The run of Mixtral 8x22B with Wikipedia context was not successfully submitted to batch 3, we might have overlooked to upload them..

For the exact list answer format in Phase B, Mixtral 8x7B with 10-shot learning took first place in batch 2 while being on place 22 of 39 in batch 1 where gpt-3-5-turbo with 10-shot learning was our best-performing one as can be seen in Table 13.

For batches 3 & 4 both Claude 3 Opus and Mixtral 8x7B performed worse with additional Wikipedia context across batches. In batch 3 our best-performing system was Mixtral 8x7B with 10-shot learning and in batch 4 it was Claude 3 Opus with 10-shot learning.

The costs for completing runs in Phase B were lower than in Phase A and the runs were faster because we did not have to do snippet extraction for 50 documents per question, times the number of few-shot examples, we therefore were able to also do 10-shot learning with Claude 3 Opus and the fine-tuned version of GPT-3.5-turbo. The processing with Mixtral 8x7B via the Fireworks.ai API only took around 30 seconds for plain 10-shot examples and around 2 minutes for 10-shot examples and additional Wikipedia context.

We also submitted ideal answers for Task B and A+, but do not report on the preliminary results here, as the official judging metric for this answer type is based on the manual judgements that are not available yet.