-
InstructExcel: A Benchmark for Natural Language Instruction in Excel
Authors:
Justin Payan,
Swaroop Mishra,
Mukul Singh,
Carina Negreanu,
Christian Poelitz,
Chitta Baral,
Subhro Roy,
Rasika Chakravarthy,
Benjamin Van Durme,
Elnaz Nouri
Abstract:
With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets. This work investigates whether LLMs can generate code (Excel OfficeScripts, a TypeScript API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale be…
▽ More
With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets. This work investigates whether LLMs can generate code (Excel OfficeScripts, a TypeScript API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale benchmark, InstructExcel, created by leveraging the 'Automate' feature in Excel to automatically generate OfficeScripts from users' actions. Our benchmark includes over 10k samples covering 170+ Excel operations across 2,000 publicly available Excel spreadsheets. Experiments across various zero-shot and few-shot settings show that InstructExcel is a hard benchmark for state of the art models like GPT-4. We observe that (1) using GPT-4 over GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can help improve performance on this benchmark.
△ Less
Submitted 22 October, 2023;
originally announced October 2023.
-
Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities
Authors:
Cyrus Cousins,
Justin Payan,
Yair Zick
Abstract:
A successful peer review process requires that qualified and interested reviewers are assigned to each paper. Most automated reviewer assignment approaches estimate a real-valued affinity score for each paper-reviewer pair that acts as a proxy for the quality of the match, and then assign reviewers to maximize the sum of affinity scores. Most affinity score estimation methods are inherently noisy:…
▽ More
A successful peer review process requires that qualified and interested reviewers are assigned to each paper. Most automated reviewer assignment approaches estimate a real-valued affinity score for each paper-reviewer pair that acts as a proxy for the quality of the match, and then assign reviewers to maximize the sum of affinity scores. Most affinity score estimation methods are inherently noisy: reviewers can only bid on a small number of papers, and textual similarity models and subject-area matching are inherently noisy estimators. Current paper assignment systems are not designed to rigorously handle noise in the peer-review matching market. In this work, we assume paper-reviewer affinity scores are located in or near a high-probability region called an uncertainty set. We maximize the worst-case sum of scores for a reviewer assignment over the uncertainty set. We demonstrate how to robustly maximize the sum of scores across various classes of uncertainty sets, avoiding potentially serious mistakes in assignment. Our general approach can be used to integrate a large variety of paper-reviewer affinity models into reviewer assignment, opening the door to a much more robust peer review process.
△ Less
Submitted 4 September, 2023; v1 submitted 25 January, 2023;
originally announced January 2023.
-
Graphical House Allocation
Authors:
Hadi Hosseini,
Justin Payan,
Rik Sengupta,
Rohit Vaish,
Vignesh Viswanathan
Abstract:
The classical house allocation problem involves assigning $n$ houses (or items) to $n$ agents according to their preferences. A key criterion in such problems is satisfying some fairness constraints such as envy-freeness. We consider a generalization of this problem wherein the agents are placed along the vertices of a graph (corresponding to a social network), and each agent can only experience e…
▽ More
The classical house allocation problem involves assigning $n$ houses (or items) to $n$ agents according to their preferences. A key criterion in such problems is satisfying some fairness constraints such as envy-freeness. We consider a generalization of this problem wherein the agents are placed along the vertices of a graph (corresponding to a social network), and each agent can only experience envy towards its neighbors. Our goal is to minimize the aggregate envy among the agents as a natural fairness objective, i.e., the sum of all pairwise envy values over all edges in a social graph.
When agents have identical and evenly-spaced valuations, our problem reduces to the well-studied problem of linear arrangements. For identical valuations with possibly uneven spacing, we show a number of deep and surprising ways in which our setting is a departure from this classical problem. More broadly, we contribute several structural and computational results for various classes of graphs, including NP-hardness results for disjoint unions of paths, cycles, stars, or cliques, and fixed-parameter tractable (and, in some cases, polynomial-time) algorithms for paths, cycles, stars, cliques, and their disjoint unions. Additionally, a conceptual contribution of our work is the formulation of a structural property for disconnected graphs that we call separability which results in efficient parameterized algorithms for finding optimal allocations.
△ Less
Submitted 18 September, 2023; v1 submitted 3 January, 2023;
originally announced January 2023.
-
Relaxations of Envy-Freeness Over Graphs
Authors:
Justin Payan,
Rik Sengupta,
Vignesh Viswanathan
Abstract:
When allocating a set of indivisible items among agents, the ideal condition of envy-freeness cannot always be achieved. Envy-freeness up to any good (EFX), and envy-freeness with $k$ hidden items (HEF-$k$) are two very compelling relaxations of envy-freeness, which remain elusive in many settings. We study a natural relaxation of these two fairness constraints, where we place the agents on the ve…
▽ More
When allocating a set of indivisible items among agents, the ideal condition of envy-freeness cannot always be achieved. Envy-freeness up to any good (EFX), and envy-freeness with $k$ hidden items (HEF-$k$) are two very compelling relaxations of envy-freeness, which remain elusive in many settings. We study a natural relaxation of these two fairness constraints, where we place the agents on the vertices of an undirected graph, and only require that our allocations satisfy the EFX (resp. HEF) constraint on the edges of the graph. We refer to these allocations as graph-EFX (resp. graph-HEF) or simply $G$-EFX (resp. $G$-HEF) allocations. We show that for any graph $G$, there always exists a $G$-HEF-$k$ allocation of goods, where $k$ is the size of a minimum vertex cover of $G$, and that this is essentially tight. We show that $G$-EFX allocations of goods exist for three different classes of graphs -- two of them generalizing the star $K_{1, n-1}$ and the third generalizing the three-edge path $P_4$. Many of these results extend to allocations of chores as well. Overall, we show several natural settings in which the graph structure helps obtain strong fairness guarantees. Finally, we evaluate an algorithm using problem instances from Spliddit to show that $G$-EFX allocations appear to exist for paths $P_n$, pointing the way towards showing EFX for even broader families of graphs.
△ Less
Submitted 3 January, 2023; v1 submitted 16 February, 2022;
originally announced February 2022.
-
Towards Realistic Single-Task Continuous Learning Research for NER
Authors:
Justin Payan,
Yuval Merhav,
He Xie,
Satyapriya Krishna,
Anil Ramakrishna,
Mukund Sridhar,
Rahul Gupta
Abstract:
There is an increasing interest in continuous learning (CL), as data privacy is becoming a priority for real-world machine learning applications. Meanwhile, there is still a lack of academic NLP benchmarks that are applicable for realistic CL settings, which is a major challenge for the advancement of the field. In this paper we discuss some of the unrealistic data characteristics of public datase…
▽ More
There is an increasing interest in continuous learning (CL), as data privacy is becoming a priority for real-world machine learning applications. Meanwhile, there is still a lack of academic NLP benchmarks that are applicable for realistic CL settings, which is a major challenge for the advancement of the field. In this paper we discuss some of the unrealistic data characteristics of public datasets, study the challenges of realistic single-task continuous learning as well as the effectiveness of data rehearsal as a way to mitigate accuracy loss. We construct a CL NER dataset from an existing publicly available dataset and release it along with the code to the research community.
△ Less
Submitted 27 October, 2021;
originally announced October 2021.
-
I Will Have Order! Optimizing Orders for Fair Reviewer Assignment
Authors:
Justin Payan,
Yair Zick
Abstract:
We present fast, fair, flexible, and welfare efficient algorithms for assigning reviewers to submitted conference papers. Our approaches extend picking sequence mechanisms, standard tools from the fair allocation literature to ensure approximate envy-freeness (typically envy-freeness up to one item, or EF1). However, fairness often comes at the cost of decreased efficiency. To overcome this challe…
▽ More
We present fast, fair, flexible, and welfare efficient algorithms for assigning reviewers to submitted conference papers. Our approaches extend picking sequence mechanisms, standard tools from the fair allocation literature to ensure approximate envy-freeness (typically envy-freeness up to one item, or EF1). However, fairness often comes at the cost of decreased efficiency. To overcome this challenge, we carefully select approximately optimal picking sequence orders. Applying a relaxation of submodularity, $γ$-weak submodularity, we show our Greedy Reviewer Round Robin (GRRR) approach is EF1 and yields a ${(1+γ)}$-approximation to the maximum welfare attainable by a round-robin picking sequence mechanism under any order. We present a weighted picking sequence mechanism called FairSequence that targets the Weighted EF1 criterion to offer fairness in a more general setting. Using data from three conferences, we show that FairSequence runs an order of magnitude faster and provides approximate envy-freeness guarantees that are violated by existing approaches. Its simple design also makes it very flexible to new assignment constraints. FairSequence is available in the OpenReview conference management platform, giving conference organizers access to faster reviewer assignment with high welfare and envy-freeness guarantees.
△ Less
Submitted 14 February, 2024; v1 submitted 4 August, 2021;
originally announced August 2021.
