-
LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
Authors:
Arash Gholami Davoodi,
Seyed Pouyan Mousavi Davoudi,
Pouya Pezeshkpour
Abstract:
Large language models (LLMs) demonstrate impressive capabilities in mathematical reasoning. However, despite these achievements, current evaluations are mostly limited to specific mathematical topics, and it remains unclear whether LLMs are genuinely engaging in reasoning. To address these gaps, we present the Mathematical Topics Tree (MaTT) benchmark, a challenging and structured benchmark that o…
▽ More
Large language models (LLMs) demonstrate impressive capabilities in mathematical reasoning. However, despite these achievements, current evaluations are mostly limited to specific mathematical topics, and it remains unclear whether LLMs are genuinely engaging in reasoning. To address these gaps, we present the Mathematical Topics Tree (MaTT) benchmark, a challenging and structured benchmark that offers 1,958 questions across a wide array of mathematical subjects, each paired with a detailed hierarchical chain of topics. Upon assessing different LLMs using the MaTT benchmark, we find that the most advanced model, GPT-4, achieved a mere 54\% accuracy in a multiple-choice scenario. Interestingly, even when employing Chain-of-Thought prompting, we observe mostly no notable improvement. Moreover, LLMs accuracy dramatically reduced by up to 24.2 percentage point when the questions were presented without providing choices. Further detailed analysis of the LLMs' performance across a range of topics showed significant discrepancy even for closely related subtopics within the same general mathematical area. In an effort to pinpoint the reasons behind LLMs performances, we conducted a manual evaluation of the completeness and correctness of the explanations generated by GPT-4 when choices were available. Surprisingly, we find that in only 53.3\% of the instances where the model provided a correct answer, the accompanying explanations were deemed complete and accurate, i.e., the model engaged in genuine reasoning.
△ Less
Submitted 7 June, 2024;
originally announced June 2024.
-
ForestDSH: A Universal Hash Design for Discrete Probability Distributions
Authors:
Arash Gholami Davoodi,
Sean Chang,
Hyun Gon Yoo,
Anubhav Baweja,
Mihir Mongia,
Hosein Mohimani
Abstract:
In this paper, we consider the problem of classification of $M$ high dimensional queries $y^1,\cdots,y^M\in B^S$ to $N$ high dimensional classes $x^1,\cdots,x^N\in A^S$ where $A$ and $B$ are discrete alphabets and the probabilistic model that relates data to the classes $P(x,y)$ is known. This problem has applications in various fields including the database search problem in mass spectrometry. Th…
▽ More
In this paper, we consider the problem of classification of $M$ high dimensional queries $y^1,\cdots,y^M\in B^S$ to $N$ high dimensional classes $x^1,\cdots,x^N\in A^S$ where $A$ and $B$ are discrete alphabets and the probabilistic model that relates data to the classes $P(x,y)$ is known. This problem has applications in various fields including the database search problem in mass spectrometry. The problem is analogous to the nearest neighbor search problem, where the goal is to find the data point in a database that is the most similar to a query point. The state of the art method for solving an approximate version of the nearest neighbor search problem in high dimensions is locality sensitive hashing (LSH). LSH is based on designing hash functions that map near points to the same buckets with a probability higher than random (far) points. To solve our high dimensional classification problem, we introduce distribution sensitive hashes that map jointly generated pairs $(x,y)\sim P$ to the same bucket with probability higher than random pairs $x\sim P^A$ and $y\sim P^B$, where $P^A$ and $P^B$ are the marginal probability distributions of $P$. We design distribution sensitive hashes using a forest of decision trees and we show that the complexity of search grows with $O(N^{λ^*(P)})$ where $λ^*(P)$ is expressed in an analytical form. We further show that the proposed hashes perform faster than state of the art approximate nearest neighbor search methods for a range of probability distributions, in both theory and simulations. Finally, we apply our method to the spectral library search problem in mass spectrometry, and show that it is an order of magnitude faster than the state of the art methods.
△ Less
Submitted 22 June, 2020; v1 submitted 11 May, 2019;
originally announced May 2019.
-
Degrees of Freedom Region of the $(M,N_1,N_2)$ MIMO Broadcast Channel with Partial CSIT: An Application of Sum-set Inequalities Based on Aligned Image Sets
Authors:
Arash Gholami Davoodi,
Syed A. Jafar
Abstract:
The degrees of freedom (DoF) region is characterized for the $2$-user multiple input multiple output (MIMO) broadcast channel (BC), where the transmitter is equipped with $M$ antennas, the two receivers are equipped with $N_1$ and $N_2$ antennas, and the levels of channel state information at the transmitter (CSIT) for the two users are parameterized by $β_1, β_2$, respectively. The achievability…
▽ More
The degrees of freedom (DoF) region is characterized for the $2$-user multiple input multiple output (MIMO) broadcast channel (BC), where the transmitter is equipped with $M$ antennas, the two receivers are equipped with $N_1$ and $N_2$ antennas, and the levels of channel state information at the transmitter (CSIT) for the two users are parameterized by $β_1, β_2$, respectively. The achievability of the DoF region was established by Hao, Rassouli and Clerckx, but no proof of optimality was heretofore available. The proof of optimality is provided in this work with the aid of sum-set inequalities based on the aligned image sets (AIS) approach.
△ Less
Submitted 17 January, 2019;
originally announced January 2019.
-
Optimality of Simple Layered Superposition Coding in the 3 User MISO BC with Finite Precision CSIT
Authors:
Arash Gholami Davoodi,
Syed Ali Jafar
Abstract:
We study the $K=3$ user multiple input single output (MISO) broadcast channel (BC) with $M=3$ antennas at the transmitter and $1$ antenna at each receiver, from the generalized degrees of freedom (GDoF) perspective, under the assumption that the channel state information at the transmitter (CSIT) is limited to finite precision. In particular, our goal is to identify a parameter regime where a simp…
▽ More
We study the $K=3$ user multiple input single output (MISO) broadcast channel (BC) with $M=3$ antennas at the transmitter and $1$ antenna at each receiver, from the generalized degrees of freedom (GDoF) perspective, under the assumption that the channel state information at the transmitter (CSIT) is limited to finite precision. In particular, our goal is to identify a parameter regime where a simple layered superposition (SLS) coding scheme achieves the entire GDoF region. With $α_{ij}$ representing the channel strength parameter for the link from the $j^{th}$ antenna of the transmitter to the $i^{th}$ receiver, we prove that SLS is GDoF optimal without the need for time-sharing if $\max(α_{ki},α_{im})\leqα_{ii}$ and $α_{ki}+α_{im}\leα_{ii}+α_{km}$ for all $i,k\in[3],m\in[M]$. The GDoF region under this condition is a convex polyhedron. The result generalizes to arbitrary $M\geq 3$.
△ Less
Submitted 14 May, 2018; v1 submitted 23 January, 2018;
originally announced January 2018.
-
$K$-User Symmetric M$\times$N MIMO Interference Channel under Finite Precision CSIT: A GDoF perspective
Authors:
Arash Gholami Davoodi,
Syed Ali Jafar
Abstract:
Generalized Degrees of Freedom (GDoF) are characterized for the symmetric $K$-user Multiple Input Multiple Output (MIMO) Interference Channel (IC) under the assumption that the channel state information at the transmitters (CSIT) is limited to finite precision. In this symmetric setting, each transmitter is equipped with $M$ antennas, each receiver is equipped with $N$ antennas, each desired chann…
▽ More
Generalized Degrees of Freedom (GDoF) are characterized for the symmetric $K$-user Multiple Input Multiple Output (MIMO) Interference Channel (IC) under the assumption that the channel state information at the transmitters (CSIT) is limited to finite precision. In this symmetric setting, each transmitter is equipped with $M$ antennas, each receiver is equipped with $N$ antennas, each desired channel (i.e., a channel between a transmit antenna and a receive antenna belonging to the same user) has strength $\sim P$, while each undesired channel has strength $\sim P^α$, where $P$ is a nominal SNR parameter. The result generalizes a previous GDoF characterization for the SISO setting $(M=N=1)$ and is enabled by a significant extension of the Aligned Image Sets bound that is broadly useful. GDoF per user take the form of a $W$-curve with respect to $α$ for fixed values of $M$ and $N$. Under finite precision CSIT, in spite of the presence of multiple antennas, all the benefits of interference alignment are lost.
△ Less
Submitted 31 October, 2017;
originally announced November 2017.
-
Network Coherence Time Matters - Aligned Image Sets and the Degrees of Freedom of Interference Networks with Finite Precision CSIT and Perfect CSIR
Authors:
Arash Gholami Davoodi,
Syed Ali Jafar
Abstract:
This work obtains the first bound that is provably sensitive to network coherence time, i.e., coherence time in an interference network where all channels experience the same coherence patterns. This is accomplished by a novel adaptation of the aligned image sets bound, and settles various open problems noted previously by Naderi and Avestimehr and by Gou et al. For example, a necessary and suffic…
▽ More
This work obtains the first bound that is provably sensitive to network coherence time, i.e., coherence time in an interference network where all channels experience the same coherence patterns. This is accomplished by a novel adaptation of the aligned image sets bound, and settles various open problems noted previously by Naderi and Avestimehr and by Gou et al. For example, a necessary and sufficient condition is obtained for the optimality of 1/2 DoF per user in a partially connected interference network where the channel state information at the receivers (CSIR) is perfect, the channel state information at the transmitters (CSIT) is instantaneous but limited to finite precision, and the network coherence time is T_c= 1. The surprising insight that emerges is that even with perfect CSIR and instantaneous finite precision CSIT, network coherence time matters, i.e., it has a DoF impact.
△ Less
Submitted 8 May, 2017;
originally announced May 2017.
-
Aligned Image Sets and the Generalized Degrees of Freedom of Symmetric MIMO Interference Channel with Partial CSIT
Authors:
Arash Gholami Davoodi,
Syed A. Jafar
Abstract:
The generalized degrees of freedom (GDoF) of the two user symmetric multiple input multiple output (MIMO) interference channel (IC) are characterized as a function of the channel strength levels and the level of channel state information at the transmitters (CSIT). In this symmetric setting, each transmitter is equipped with M antennas, each receiver is equipped with N antennas, and both cross lin…
▽ More
The generalized degrees of freedom (GDoF) of the two user symmetric multiple input multiple output (MIMO) interference channel (IC) are characterized as a function of the channel strength levels and the level of channel state information at the transmitters (CSIT). In this symmetric setting, each transmitter is equipped with M antennas, each receiver is equipped with N antennas, and both cross links have the same strength parameter $α$ and the same channel uncertainty parameter $β$. The main challenge resides in the proof of the outer bound which is accomplished by a generalization of the aligned image sets approach.
△ Less
Submitted 1 May, 2017;
originally announced May 2017.
-
Sum-set Inequalities from Aligned Image Sets: Instruments for Robust GDoF Bounds
Authors:
Arash Gholami Davoodi,
Syed A. Jafar
Abstract:
We present sum-set inequalities specialized to the generalized degrees of freedom (GDoF) framework. These are information theoretic lower bounds on the entropy of bounded density linear combinations of discrete, power-limited dependent random variables in terms of the joint entropies of arbitrary linear combinations of new random variables that are obtained by power level partitioning of the origi…
▽ More
We present sum-set inequalities specialized to the generalized degrees of freedom (GDoF) framework. These are information theoretic lower bounds on the entropy of bounded density linear combinations of discrete, power-limited dependent random variables in terms of the joint entropies of arbitrary linear combinations of new random variables that are obtained by power level partitioning of the original random variables. These bounds generalize the aligned image sets approach, and are useful instruments to obtain GDoF characterizations for wireless networks, especially with multiple antenna nodes, subject to arbitrary channel strength and channel uncertainty levels. To demonstrate the utility of these bounds, we consider a non-trivial instance of wireless networks - a two user interference channel with different number of antennas at each node, and different levels of partial channel knowledge available to the transmitters. We obtain tight GDoF characterization for specific instance of this channel with the aid of sum-set inequalities.
△ Less
Submitted 24 August, 2017; v1 submitted 3 March, 2017;
originally announced March 2017.
-
GDoF of the MISO BC: Bridging the Gap between Finite Precision and Perfect CSIT
Authors:
Arash Gholami Davoodi,
Bofeng Yuan,
Syed A. Jafar
Abstract:
For the $K=2$ user MISO BC, i.e., the wireless broadcast channel where a transmitter equipped with $K=2$ antennas sends independent messages to $K=2$ receivers each of which is equipped with a single antenna, the sum generalized degrees of freedom (GDoF) are characterized for arbitrary channel strength and channel uncertainty levels for each of the channel coefficients. The result is extended to…
▽ More
For the $K=2$ user MISO BC, i.e., the wireless broadcast channel where a transmitter equipped with $K=2$ antennas sends independent messages to $K=2$ receivers each of which is equipped with a single antenna, the sum generalized degrees of freedom (GDoF) are characterized for arbitrary channel strength and channel uncertainty levels for each of the channel coefficients. The result is extended to $K>2$ users under additional restrictions which include the assumption of symmetry.
△ Less
Submitted 26 August, 2016; v1 submitted 5 February, 2016;
originally announced February 2016.
-
Generalized Degrees of Freedom of the Symmetric K-User Interference Channel under Finite Precision CSIT
Authors:
Arash Gholami Davoodi,
Syed A. Jafar
Abstract:
The generalized degrees of freedom (GDoF) characterization of the symmetric K-user interference channel is obtained under finite precision channel state information at the transmitters (CSIT). The symmetric setting is where each cross channel is capable of carrying degrees of freedom (DoF) while each direct channel is capable of carrying 1 DoF. Remarkably, under finite precision CSIT the symmetric…
▽ More
The generalized degrees of freedom (GDoF) characterization of the symmetric K-user interference channel is obtained under finite precision channel state information at the transmitters (CSIT). The symmetric setting is where each cross channel is capable of carrying degrees of freedom (DoF) while each direct channel is capable of carrying 1 DoF. Remarkably, under finite precision CSIT the symmetric K-user interference channel loses all the GDoF benefits of interference alignment. The GDoF per user diminish with the number of users everywhere except in the very strong (optimal for every receiver to decode all messages) and very weak (optimal to treat all interference as noise) interference regimes. The result stands in sharp contrast to prior work on the symmetric setting under perfect CSIT, where the GDoF per user remain undiminished due to interference alignment. The result also stands in contrast to prior work on a subclass of asymmetric settings under finite precision CSIT, i.e., the topological interference management problem, where interference alignment plays a crucial role and provides substantial GDoF benefits.
△ Less
Submitted 24 January, 2016;
originally announced January 2016.
-
Aligned Image Sets under Channel Uncertainty: Settling a Conjecture by Lapidoth, Shamai and Wigger on the Collapse of Degrees of Freedom under Finite Precision CSIT
Authors:
Arash Gholami Davoodi,
Syed A. Jafar
Abstract:
A conjecture made by Lapidoth, Shamai and Wigger at Allerton 2005 (also an open problem presented at ITA 2006) states that the DoF of a 2 user broadcast channel, where the transmitter is equipped with 2 antennas and each user is equipped with 1 antenna, must collapse under finite precision CSIT. In this work we prove that the conjecture is true in all non-degenerate settings (e.g., where the proba…
▽ More
A conjecture made by Lapidoth, Shamai and Wigger at Allerton 2005 (also an open problem presented at ITA 2006) states that the DoF of a 2 user broadcast channel, where the transmitter is equipped with 2 antennas and each user is equipped with 1 antenna, must collapse under finite precision CSIT. In this work we prove that the conjecture is true in all non-degenerate settings (e.g., where the probability density function of unknown channel coefficients exists and is bounded). The DoF collapse even when perfect channel knowledge for one user is available to the transmitter. This also settles a related recent conjecture by Tandon et al. The key to our proof is a bound on the number of codewords that can cast the same image (within noise distortion) at the undesired receiver whose channel is subject to finite precision CSIT, while remaining resolvable at the desired receiver whose channel is precisely known by the transmitter. We are also able to generalize the result along two directions. First, if the peak of the probability density function is allowed to scale as O(P^(α/2)), representing the concentration of probability density (improving CSIT) due to, e.g., quantized feedback at rate (α/2)\log(P), then the DoF are bounded above by 1+α, which is also achievable under quantized feedback. Second, we generalize the result to the K user broadcast channel with K antennas at the transmitter and a single antenna at each receiver. Here also the DoF collapse under non-degenerate channel uncertainty. The result directly implies a collapse of DoF to unity under non-degenerate channel uncertainty for the general K-user interference and MxN user X networks as well.
△ Less
Submitted 6 March, 2014;
originally announced March 2014.
-
Optimum Power Allocations for Fading Decode-and-Forward Relay Channel
Authors:
Arash Gholami Davoodi,
Mohammad Javad Emadi,
Mohammad Reza Aref
Abstract:
In this paper, for a fading decode-and-forward full-duplex relay channel, we analytically derive optimum power allocations. Individual power constraints for the source and the relay are assumed and the related optimization problem is analyzed for two scenarios. First, optimization is taken over the source power, the relay power, and the correlation coefficient between the transmitted signals of th…
▽ More
In this paper, for a fading decode-and-forward full-duplex relay channel, we analytically derive optimum power allocations. Individual power constraints for the source and the relay are assumed and the related optimization problem is analyzed for two scenarios. First, optimization is taken over the source power, the relay power, and the correlation coefficient between the transmitted signals of the source and the relay. Then, for a fixed value of correlation coefficient, the optimization problem is analyzed. It is also proven that the optimization problems are convex for these two scenarios. Finally, implications of theoretical results are discussed through simulations for each scenario.
△ Less
Submitted 6 February, 2012;
originally announced February 2012.
