LOTUS: Enabling Semantic Queries with LLMs Over Tables of Unstructured and Structured Data

LOTUS is designed to seamlessly extend the relational model. Each row in the table, represents a logical entity. Table columns may contain either structured data or unstructured fields with free-form natural-language text. LOTUS’ semantic-relational operators can take both of these data-types as inputs. Figure 3 shows an example table, where each rows represents an ArXiv paper document, with fields for the paper’s title, ArXiv URL, abstract, ArXiv domain categories, and the publication date. These columns are then passed as parameters to semantic-relational operators, such as sem_search, sem_filter, sem_agg, as shown in Figure 2.

Additionally, LOTUS supports semantic similarity indices over natural-language text columns to provide optimized semantic query processing. These indices create embeddings over each document in the column to capture semantic similarity using embedding distance metrics. Semantic indexes can be created off-line and then loaded using sem_index and load_sem_index. Figure 2 provides an example program, where upon reading the ArXiv papers data from a CSV file, the programmer loads a semantic index over the abstract column. The program then repeatedly uses the semantically-indexed column in subsequent LOTUS operations, involving semantic search, filtering and aggregation.

A core principle of LOTUS is to provide users with a declarative interface that separates the user-specified, logical query plan from its underlying implementation. As such, users program with LOTUS’ semantic operators by writing parameterized natural language expressions (langex¹¹1Akin to regular expressions (regex), which specify character patterns to match in text, langex are natural language expressions to programmatically specify reasoning-based patterns over structured data and free-form text.), rather than directly prompting an underlying LM. Figure 2 shows an example of this, where the programmer provides a langex as the parameter to the sem_filter (line 18) and sem_agg (line 19) operations. Programmers write each langex in natural language text, parameterized by one or more data columns, which are indicated in the double brackets within the formatted string. The function of these expressions varies according to the semantic operator used and may represent a predicate, aggregation, comparator function, or projection in natural language. For instance, as shown in Figure 2, the langex signature of sem_filter provides a predicate that indicates a filter criteria to apply over paper titles and abstracts, while sem_agg takes a langex that provides an associative aggregation expression, which here indicates a many-to-one summarization task over paper abstracts.

Notably, these language expressions are sufficiently versatile and easy to program with, providing an intuitive and higher-level interface to the user. This makes it simple for the programmer to specify diverse and complex, multi-step query pipelines. All operator-specific prompts are automatically handled by LOTUS’ underlying query engine, and the system can leverage existing prompt optimization techniques (Khattab et al., 2023; Yuksekgonul et al., 2024) on- or off-line.

We leverage model cascades, using an efficient scoring function unique to our setting, to further optimize the filter function. Many prior works have studied model cascades for ML tasks (Viola and Jones, 2001; Yue et al., 2024; Chen et al., 2023; Kang et al., 2017, 2022), leveraging a relatively cheap and inaccurate proxy model, along with an accurate but expensive oracle model in order to reduce the inference cost by routing easy queries to the small model, and resorting to the large model where necessary. Typically, a scoring function provides confidence scores over the proxy model’s outputs, and allows the system to decide when to route examples to the larger model. Applying this paradigm to LMs introduces several challenges, including generating a reliable scoring function. Several works study this problem, suggesting to fine-tune a smaller LM to score each question along with the answer produced by the weak LLM (Chen et al., 2023), or invoking multiple sampling paths from the small LM and evaluating self-consistency to build a confidence score (Yue et al., 2024). While these methods provide general-purpose scoring functions, LOTUS’ scoring function leverages the operator semantics to provide a lighter-weight scoring function, which avoids the execution time penalty of scoring with an additional model or re-sampling from the proxy-model.

Specifically, LOTUS’ cascade scoring function leverages the binary-label output of each LM invocation. The system first batch-processes each record using the cheaper LM, then computes confidence scores by exponentiation the log-probabilities for the output LLM tokens corresponding to the True oder False answer. For records with confidence scores below the user-defined threshold, the system then passes these records in batch to the larger LM. As we show, in Section 4, leveraging model cascades in this way for semantic filtering can significantly improves the system’s efficiency while maintaining high quality results for some tasks.

Prior LM-based sorting and ranking works suggest several methods for performing LM-based comparisons. Specifically, prior works primarily study three classes of methods: point-wise ranking methods (Desai and Durrett, 2020; Drozdov et al., 2023; Liang et al., 2022; Sachan et al., 2022; Wu et al., 2024), list-wise ranking methods (Ma et al., 2023; Pradeep et al., 2023a, b; Sun et al., 2023), and pair-wise ranking methods (Shah and Wainwright, 2016; Qin et al., 2024). Point-wise methods output a relevance score for each document independently; while these methods are fully parallelizable across rows, prior work (Desai and Durrett, 2020) shows they provide poor accuracy due to difficulty in calibrating scores across prompts. We note that these methods can be implemented in LOTUS with a sem_map, but are not the focus of our semantic top-$k$ operator due to their quality limitations. By contrast, list-wise methods feed a partial list of 10 to 20 documents to the LM and prompt it to output a ranking over these, which can be aggregated using a sliding window approach. Unfortunately, prior works shows these methods are often prone to prediction failures (Sun et al., 2023) and sensitive to document ordering in the prompt (Qin et al., 2024). Pairwise-prompting methods instead offer a simple and effective approach that feeds a single pair of documents to the LM in each invocation, prompting the model to perform a comparison and output a binary label. This method has been shown to be an effective base unit for ranking with relatively high robustness to input order sensitivity (Qin et al., 2024).

To aggregate pairwise comparisons, we consider several ranking algorithms, including a quadratic sorting algorithm, a heap-based top-k algorithm and a quick-select-based top-k ranking algorithm. The quadratic ranking algorithm obtains a comparison between each pair of input documents, and uses a win rate to determine a ranking over all elements before selecting the $k$ best. In contrast, the heap-based algorithm maintains a heap of size $k$, and makes a linear pass over the data, updating the heap when more promising elements are found. Each time a new element is inserted or removed from the heap, the algorithm performs a series of sequential LM comparisons to update the data structure. Lastly the quick-select based algorithm proceeds in successive rounds, each time choosing a pivot, and comparing all other remaining elements in the document set to the pivot item to determine the rank of the pivot. Because each round is fully parallelizable, we perform these LM-based comparisons efficiently in batch before recursing in the next round.

As the core LM-based computation unit over subsets of the data, LOTUS’ current implementation uses pair-wise rankings, following recent prior work (Qin et al., 2024), which demonstrates its effectiveness. We also find this choice useful for implementing further optimizations, such as model cascades, described below. Figure 19 shows an example task-instruction prompt, which we use to implement the LM-based comparison by instructing the model to select between two documents according to the user-defined sorting criteria.

Additionally, for aggregating comparisons, LOTUS’ current implementation leverages the quick-select-based (Hoare, 1961) top-k ranking algorithm, by default. Prior work (Qin et al., 2024) studies several alternative algorithms, including the heap-based ranking implementation and the quadratic sorting algorithm. We study these alternatives in Section 4, and find that the quick-select-based algorithm offers high accuracy while also offering an efficient implementation with at least an order magnitude fewer total calls then the quadratic sorting algorithm and more opportunities for batched calls leading to lower execution time compared to a heap-based implementation. We believe future work may leverage these multiple algorithms to allow the user to declaratively trade-off between performance metrics, like accuracy, execution time, and cost.

LOTUS’ top-$k$ algorithm is amenable to several optimizations. First, the top-$k$ algorithm’s pair-wise comparisons each invoke the LM to output a binary label, which is amenable to model cascades. To implement this, we use a simple an efficient scoring procedure based on log-probabilities of the generated tokens. We describe this procedure above for sem_filter (Section 3.1.1) and apply an equivalent procedure here. In Section 4, we demonstrate this optimization allows programmers to leverage small, cheap models in conjunction with large, expensive models to obtain high accuracy results at reduced cost.

Additionally, LOTUS can leverage the semantic index to optimize pivot selection for some queries, rather than resorting to random pivot selection. This optimization is useful when there exists correlation between the rankings imposed by the user’s arbitrary sorting criteria and the rankings imposed by semantic similarity scores. In this case, LOTUS can sort the document set based on embedding distances to the user’s query, and select the $(k+\epsilon)$-th item, rather than a random item, as the first pivot. This can reduce the number of LM comparisons required by subsequent rounds in the quick-select algorithm, leading to higher query efficiency at no accuracy loss. We believe fruitful future work will automatically estimate correlation between semantic similarity scores and the user-defined ranking criteria to transparently apply this optimization.

The first join algorithm implements the nested-loop join pattern with efficient LM batch processing to maximize GPU utilization rather than naively looping over each pair of rows and invoking the LM. This yields an $O(N_{1}\cdot N_{2})$ LM call complexity, where $N_{1}$ and $N_{2}$ are the table sizes of the left and right join tables respectively. As Figure 18 shows, each LM call instructs the model to output a boolean value after evaluating the user’s natural language predicate. This quadratic join algorithm is suitable for small join tables.

Alternatively, we implement a map-search-filter join pattern which can be used to approximate the quadratic join algorithm, while incurring fewer LM calls. This algorithm first performs a semantic mapping over the left join key to the domain of the right join key. In the example provided by Figure 20, which joins paper abstracts with the datasets they use, the map step would invoke the LM over each abstract, instructing the model to output the dataset used. This step is un-grounded in the sense that the LM is not given knowledge of right join table, but may optionally leverage user-specified demonstrations. Following the semantic projection, the algorithm then leverages the semantic index and performs a similarity search over the right join table to find candidates likely to pass the predicate. In the example, this search would retrieve a list of datasets from the right table for each paper, using semantic similarity of the datasets to the LM projection output in the first step. The semantic similarity search is parameterized by $K$, which is automatically set based on the user’s specified LM call budget. Lastly, the algorithm performs a filter over the candidate pairs and outputs the joined table of tuples pairs that pass. This join algorithm partially mimics the information flow of prior work (D’Oosterlinck et al., 2024) and abstracts away intermediate steps by instead allowing users to specify an LM call budget for the join. The LM call complexity of this algorithm is $O(N_{1}\cdot K)$.

Lastly, we implement a search-filter join pattern. This algorithm first performs a semantic similarity join using a similarity index on the right join key, with the left join key embedded on the fly. In the example provided by Figure 20, this step would retrieve $K$ semantically similar datasets from the right table for each paper abstract in the left table. The batched semantic-similarity search sets the search parameter $K$ according to the user’s specified LM call budget, similar to the map-search-filter algorithm. The search results provide candidate tuple pairs, which are then evaluated in batch using the filter operation to gather the final set of rows that pass the user’s language predicate. This algorithm obtains an LM call complexity of $O(N_{1}\cdot K)$.

These join patterns offer performance tradeoffs suitable for different settings. The nested-loop algorithm offers an efficient solution when the join tables are sufficiently small, whereas the map-search-filter and search-filter patterns are suitable approximations that can apply efficiently over large tables, due to their linear LM call complexity in table size. The expected result quality of either approximation likely varies depending on the presence of predicate clustering and correlation, as defined by prior work (Patel et al., 2024) between the user’s predicate and the semantic embeddings. The search-filter pattern is likely to produce high quality results under positive query correlation, where entities that are semantically similar are also likely to pass the predicate. On the other hand, the map-search-filter pattern is likely to produce higher quality results when the predicate-embedding correlation is low using the original left join key’s embeddings as queries, but can be increased using the LM-projection over the left join key.

LOTUS’ implementation builds on and generalizes the LM-based summarization pattern studied by prior research works (Wu et al., 2021; Chang et al., 2024; Adams et al., 2023) and deployed systems (lla, [n. d.]; lan, [n. d.]). These implementations primarily leverage one of two aggregation patterns: either a fold pattern, which produces a sequential, linear pass over the data while iteratively updating an accumulated partial answer, or a hierarchical reduce pattern, which recursively aggregates the input data to produce partial answers until a single answer remains.

By default, LOTUS’ aggregation implements the hierarchical pattern, which allows for greater parallelism during query processing and has been shown to produce higher quality results for tasks like summarization in prior work (Chang et al., 2024). However, LOTUS’ partition function can be used to override the default functionality and generalizes the information flow of prior implementation by allowing users to specify arbitrary groupings over the data. Each unique group specified by the partition function will be aggregated with as few LM invocations as possible according to the model context length, before being merged with other groups. This allows the user to perform a fold pattern, as well as arbitrary user-defined patterns.

Preliminary results show that achieving high-quality semantic-aggregations is non-trivial and sensitive to the document’s input ordering and grouping. We see qualitative evidence of this in a summarization task of 50 ArXiv paper abstracts, which we show in Figure 21. We contrast the summarization results of a naive semantic aggregation (Figure 22) performed over the paper abstracts with the semantic aggregation performed using a partitioning function based on semantic similarity (Figure 23). The semantic aggregation using the partitioner first clusters documents based on semantic similarity, then aggregates each cluster before performing aggregations across clusters. As the figures show the two methods result in significantly different summaries. While the latter demonstrates qualitatively more cohesion and effectively abstracts general themes across papers, the former omits details and tends to list low-level details from individual papers rather than capturing higher-level themes. We leave a quantitative study of this to future work and believe that semantic aggregations create a rich design space for optimization.

We evaluate on FEVER (Thorne et al., 2018), a claim verification dataset. We use the development dataset, which contains about 38,000 total claims, of which we sample 500 for our evaluation. Each claim is labeled with one of three labels, ”Supported”, ”Refuted”, or ”NotEnoughInfo”, and the task is to correctly determine the label of each claim, leveraging evidence from a corpus of $~{}5.5$ million Wikipedia articles. We merge the latter two labels in to a single class, ”Not Supported”, following prior work (Chern et al., 2023) for our evaluation.

FacTool (Chern et al., 2023) is a recent research work that proposes a multi-step pipeline for fact-checking involving, claim extraction, query generation, tool querying, evidence collection, and verification. We use FactTool’s open source codebase (gai, [n. d.]) to measure its performance. FactTool’s pipeline, by default, performs retrieval with a Google Search API (cus, [n. d.]). We evaluate the pipeline with both the default retrieval API, and alternatively test with a ColBERT (Khattab and Zaharia, 2020) index over the document corpus to perform retrieval. We find that the results are similar, and we report the results using ColBERT for retrieval to hold the retriever model constant with the implemented LOTUS programs.

We compose several intuitive LOTUS programs, each in less than 50 lines of code. For each one, we use ColBERT as the retriever model for creating the semantic index, and we use Llama 70B as the primary LM, and Llama 8B and TinyLlama for cascade optimizations.

First, we compose a pipeline designed to directly re-implement FacTool’s information flow in LOTUS. Figure 24 shows the pseudocode for the LOTUS-FacTool pipeline. The two tables are shown by wiki_df, which stores the Wikipedia articles, and claim_df, which stores the claims from the FEVER dataset. After loading the semantic index to wiki_df, the pipeline first performs a semantic map over each claim to generate two search queries, which are then used to perform a semantic similarity join over the corpus of Wikipedia articles. The program then concatenates the context retrieved for each claim, and performs a sem_map to output whether the claim is true or false, along with a revised claim if the claim is false. We use the same prompts found in FacTool (gai, [n. d.]), which include 3 demonstrations for generating search queries in the first sem_map, and chain-of-thought prompting in the second sem_map.

The next program, LOTUS-fact-filter, makes a simple, single-operator modification to the LOTUS-FacTool program, replacing the semantic map at the end of the query pipeline, with LOTUS’ semantic filter operation. We use 3 demonstrations for the filter. Figure 25 shows the pseudocode for this program. We evaluate this program with and without model cascades for the semantic filter operation.

Lastly, we compose an alternative pipeline, LOTUS-fact-join. As the pseudocode shows in Figure 26, the pipeline first performs a semantic map over each claim to obtain a set of sub-claims. From this, we create the claimed_facts_df, which separates each sub-claim into a different row. Next, the pipeline uses these sub-claims to perform a semantic similarity join over the Wikipedia corpus, then performs a semantic map over each retrieved article to generate the important facts in each one. Lastly, the program performs a semantic join between the sub-claims and the facts described in the retrieved passages. If the returned table contains a supporting fact for each sub-claim, the claim is labeled as ”Supported”. For the sem_map and join operations, we use 3 demonstrations each.

Table LABEL:tab:factchecking demonstrates the powerful abstraction that LOTUS provides, allowing programmers to quickly write and test programs that compose simple operators to obtain state-of-the-art results. We report the accuracy of each benchmarked method, an estimate of lines of code (LoC), and execution time in seconds both with and without batching. We see that FacTool’s implementation offers strong accuracy performance on the FEVER datasets, however the full repository required several hundred lines of code, highlighting the development burden of building these applications without abstractions for semantic-bulk processing. By contrast, each LOTUS program offers comparable or higher accuracy, in relatively few lines of code. We also note that FacTool implements its method without batching, whereas LOTUS, by default, leverages batched LM execution for efficiency. To provide an apples-to-apples comparison, we compare FacTool’s un-batched implementation to the LOTUS programs both with and without batching.

First, we see from the Table LABEL:tab:factchecking that LOTUS-FacTool is able to reproduce the result quality and efficiency of the original method’s implementation, with 6.5 points higher accuracy and $1.2\times$ lower execution time without batching. The LOTUS-FacTool implementation with batching further decreases execution time compared to the original FacTool implementation by $10\times$. We see that the next LOTUS pipeline simply changes a single operation, switching the sem_map to a sem_filter, and maintains similar result quality to LOTUS-FacTool while further reducing execution time by $1.72\times$ in the batched implementation.

Leveraging LOTUS’ filter operation allows the programmer to further optimize the program using model cascades, which increases accuracy by 3 points and increases the batched execution time by $3.27\times$ compared to LOTUS-FacTool. Figure 27 highlights the diverse performance trade-offs presented the model cascade optimizations used in LOTUS’ filter sem_filter. The plot shows the accuracy and execution time of the LOTUS-fact-filter pipeline using a single model, either Llama 8b, Llama 70B, or TinyLlama (Zhang et al., 2024), shown by the circles. We compare this to the performance attainable using a pair of models, Llama 8B and Llama 70B, or TinyLlama and Llama70B to implement the filter casacade. We generate multiple cascade points, shown by the stars, by varying the confidence threshold used. The plot shows that this filter optimization can substantially reduce execution time, and offer diverse accuracy trade-offs, compared to implementing the pipeline with the oracle model, Llama 70B, alone.

Returning to Table LABEL:tab:factchecking, we find that the last LOTUS pipeline (search-map-join), reduces accuracy and increases execution time. Notably the performance trade-offs of different LOTUS programs are non-obvious, highlighting the need for programmable, declarative abstractions so programmers can quickly explore and iterate on their query pipelines. Each LOTUS program we described can be implemented easily in relatively few lines of code, and the best performing one offers $9.5\%$ higher accuracy and $34\times$ lower execution time than FacTool’s original, un-batched implementation.

We evaluate on the Biodex Dataset (D’Oosterlinck et al., 2023), which consists of a corpus of $65,000$ biomedical articles, and expert-created drug safety reports constructed from each article. The task is to correctly label the drug reactions experience by the patient in each medical article. We sample 250 patient articles for our evaluation. Notably, there are approximately 24,000 possible drug-reaction labels, making this task an extreme multi-label classification task. Due to the large number of possible labels, leveraging an LM to perform inference is difficult, and this setting has been studied in prior works (D’Oosterlinck et al., 2024). We show below that this task can be efficiently modeled and programmed using LOTUS’ semantic join.

We consider a simple retrieval baseline which uses an E5Model (Wang et al., 2024) as the retriever and performs a semantic-similarity join between the patient articles and the reaction labels. We show the pseudocode for this program in Figure 28.

The proposed LOTUS program performs a semantic join over the drug reaction labels and the medical articles, as shown in Figure 32. We perform the semantic join using at most 7 map demonstrations and set an LM call budget of 10,000. We use Llama-70b as the LM.

Table 3 shows that the proposed LOTUS program makes meaningful traction, obtaining high quality results on this task, while maintaining query efficiency. The table reports the rank-precision@5 (RP@5), rank-precision@10 (RP@10), following prior work (D’Oosterlinck et al., 2024), as well as execution time in seconds and the number of LM calls required for each program. Since the nested-loop join pattern is prohibitively expensive to run, we show an estimate of the execution time assuming linear scaling in the number of batched calls.

First we compare performance of the LOTUS join program implemented with the map-search-filter pattern to the baseline method, the semantic similarity join. As expected, the LOTUS program offers substantially higher quality results, with $2.27\times$ and $2.15\times$ higher RP@5 and RP@10 respectively compared to the retrieval-based similarity join. This highlights the effectiveness of leveraging LMs over the data for complex reasoning-based tasks. We informally compare these accuracy results to recent work (D’Oosterlinck et al., 2024) which composes a multi-step DSPy (Khattab et al., 2023) program compiled using Llama-2-7b-chat as the student LM and GPT-3.5-turbo as the teacher LM to perform a semantic mapping, followed by a re-ranking step using GPT-4 turbo. D’Oosterlinck et al. report 24.73 RP@5 and 27.67 RP@10 for the compiled program, representing comparable result quality to LOTUS’ program, although notably the LOTUS program was not compiled with a prompt optimization system.

We now consider several interesting performance trade-offs presented in the semantic join algorithm. We compare LOTUS map-search-filter join pattern to the naive nested-loop join algorithm and the search-filter join pattern shown in Table 3. We see that nested-loop join pattern, which involves a quadratic LM budget of over 6 million LM calls, is untenable and prohibitively costly. By contrast, the search-filter and map-search-filter pattern substantially reduce the LM call budget of the naive algorithm by $800\times$, using an approximation. While these two approximation patterns have similar efficiency, according to execution time and the number of LM calls, interestingly, they offer substantially different result quality. Specifically, the map-search-filter pattern offers $55\%$ higher RP@5 and $38\%$ higher RP@10 compared to the retrieve-filter pattern on this task. These results highlight the unique opportunities bulk-semantic processing pipelines present for designing new algorithms and optimizations.

We evaluate LOTUS’ performance on the search and ranking task using three datasets, including BEIR’s SciFact test set (Thakur et al., 2021), a widely used benchmark for retrieval and re-ranking, as well as two new benchmarks, CIFAR-bench, and HellaSwag-bench, which we generate to evaluate more complex, reasoning-based ranking criteria over the data. For each, we report nDCG@10, as well as the execution time (ET) in seconds.

The SciFact dataset consists of a set of scientific claims and a corpus of articles, where the task is to rank articles by relevance given each scientific claim. We sample 300 scientific facts for our evaluation, and report the average ranking execution time across these samples.

While the SciFact dataset provides a sorting task based on a simple relevance criterion, our newly proposed benchmarks provide a more complex sorting criteria over a corpus of paper abstracts. Specifically, the ranking task is to find the papers that report the highest accuracy on CIFAR-10 and HellaSwag in CIFAR-bench and HellaSwag-bench respectively. To generate CIFAR-bench, we took 100 abstracts from the Papers with Code Dataset (pap, [n. d.]) that state performance on CIFAR-10 in the abstract, and we and manually labeled their accuracy to obtain the top-10. We then synthetically generated HellaSwag-bench by prompting Llama-70B to create 200 paper abstracts, each with a specified accuracy value, randomly sampled from $0-100\%$. This setup allows us to evaluate LOTUS’ LM-based sorting algorithms on a task with objective ground truth. We note that an alternative approach to these tasks could efficiently leverage sem_map to extract accuracy values on abstracts from either dataset, then perform a structured sort. However, our evaluation focuses on assessing the semantic ranking capabilities of LOTUS’ top-k algorithms, and we find this benchmark useful for understanding performance trade-offs, which may likely generalize to a wider set of reasoning-based sorting queries, such as ”which paper makes the most outrageous claim”, for which ground truth is less objective to evaluate. For CIFAR-bench and HellaSwag-bench we report results for $n=20$ trials of the ranking task, at temperature $t=0.7$, similar to prior works (Khattab et al., 2023)

We consider two simple baselines. The first baseline performs semantic search, as shown in Figure 30, using the E5Model (Wang et al., 2024) for retrieval. The second baseline performs search with re-ranking, as the pseudocode shows in Figure 31, using the E5Model for retrieval and the MixedBread cross-encoder (noa, 2024) for re-ranking.

The proposed LOTUS program performs a semantic top-$k$ over the documents, as shown in Figure 32, which shows example pseudocode for the CIFAR-bench dataset. The langex for the semantic top-$k$ on SciFact sorts based on relevance to the given claim, while the langex for the CIFAR-bench and HellaSwag-bench datasets sort abstracts by accuracy performance on the respective datasets, as shown in the figure. We note that for the SciFact dataset, we perform a semantic search using the E5Model as the retriever to obtain 100 articles, before ranking them with the sem_topk. We report results using both Llama-70B and GPT-4o as the LM.

Tables LABEL:tab:ranking_scifact and 5 demonstrate the effectiveness of LOTUS’ semantic top-$k$ operator for complex search tasks. In addition, Table 6 and Figure 33 highlight the rich implementation design space that this task presents. We walk through several note-worthy findings.

We first turn our attention to Table LABEL:tab:ranking_scifact, which presents the results for each bench-marked method on the SciFact dataset, which uses a relevance-based sorting criterion. As expected, both semantic search programs with and without re-ranking present strong baselines. Specifically, the re-ranker model, which is a supervised model trained specifically for the task of relevance-based ranking, increases nDCG@10 by 3 percentage points, while trading off query efficiency compared to the semantic search baseline. Notably, the unsupervised LM-based LOTUS programs outperform the supervised re-ranker’s result quality. The table shows LOTUS semantic top-k program with Llama-70B and GPT-4o, which outperform the semantic search baseline by 6 and 9 percentage points respectively. The LOTUS programs with Llama-70B and GPT-4o also outperform the re-ranker by 3 and 5 points respectively, improving upon the quality of the supervised baseline. As expected, the improved result quality comes at a trade-off to query efficiency due to the cost of LM calls. Notably, LOTUS’ versatility allows programmers to easily compose each of these query pipeline and trade-off result quality and efficiency depending on application-specific requirements.

Turning our attention to Table 5, we study LOTUS’ generality in supporting arbitrary language-based ranking criteria over the dataset. On the CIFAR-bench and HellaSwag-bench datasets, which use a complex sorting criteria, we see that both semantic search baselines with and without re-ranking provide poor result quality with consistently low nDCG@10. The LOTUS program, using a semantic top-$k$ with Llama 70B, acheives significant accuracy gains, with 49.4 and 44.8 points higher nDCG@10 than the best performing baseline on CIFAR-bench and HellaSwag respectively. These accuracy gains reflect the powerful reasoning capabilities of LMs efficiently orchestrated over the data. As expected, these significant accuracy gains come at an increase to execution time.

We now analyze the efficiency of LOTUS semantic top-$k$ implementation along with it’s proposed optimizations. Table 6 compares several semantic top-$k$ algorithms, namely a quick-select top-$k$ algorithm, a quick-select top-$k$ that leverages the similarity index for pivot selection, a heap-based top-$k$ algorithm, and a quadratic sorting algorithm. First, we see that the quadratic algorithm, which performs an LM comparison between each pair of input documents, offers consistently high result quality across each dataset. However, this method is prohibitively expensive, requiring $16-30\times$ more LM calls and over $10\times$ higher execution time than the alternative implementations. The heap top-$k$ and quick-select top-$k$ methods offer comparable result quality, but with interesting trade-offs in query efficiency. Notably the quick-select top-$k$ method offers $1.67-2.24\times$ lower execution time than the heap-based sorting method across all datasets, despite requiring more LM calls in some cases. This is because the quick-select top-$k$ implementation allows for efficient batch-processing in each round of the algorithm, whereas the heap-based top-$k$ incurs sequential LM calls during heap updates. For this reason, our current implementation leverages the quick-select top-$k$ algorithm, although we envision future iterations may leverage multiple algorithms and allow the user to declaratively trade-off accuracy, query throughput, and cost.

In addition to providing an efficient top-$k$ algorithm, the table also demonstrates the use of LOTUS’ similarity index for optimizing top-$k$ query performance. The quick-select top-$k$ algorithm optimized with the semantic similarity index for pivot selection demonstrates $1.2\times$ lower execution time at no accuracy loss on SciFact, where the ranking criteria correlates likely with semantic similarity. On the other hand, for the CIFAR-bench and HellaSwag-bench datasets, where the ranking criteria does not correlate with semantic similarity, we see that the similarity index has no significant impact on the accuracy or efficiency top-$k$ performance.

Lastly, we analyze the impact of LOTUS’ cascade optimization applied to the quickselect top-$k$ implementation. Figure 33 compares the nDCG and execution time of implementing the operator with a single model, either Llama 8B or Llama 70B, to the implementation that leverages these models together using model cascades. We vary the confidence threshold of the cascade to generate several points in the trade-off space. We find that the cascade optimization offers diverse performance trade-offs, which can outperform the single-model oracle baseline, which uses Llama-70B. For instance, one cascade along the Pareto-frontier improves accuracy of the Llama-70B baseline by $3\%$, while reducing execution time by $1.8\times$, demonstrating substantial opportunities for automatically optimizing LOTUS’ semantic query pipelines.