Showing 1–15 of 15 results for author: Sinha, D

Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters

Authors: Manikanta Loya, Divya Anand Sinha, Richard Futrell

Abstract: The advancement of Large Language Models (LLMs) has led to their widespread use across a broad spectrum of tasks including decision making. Prior studies have compared the decision making abilities of LLMs with those of humans from a psychological perspective. However, these studies have not always properly accounted for the sensitivity of LLMs' behavior to hyperparameters and variations in the pr… ▽ More The advancement of Large Language Models (LLMs) has led to their widespread use across a broad spectrum of tasks including decision making. Prior studies have compared the decision making abilities of LLMs with those of humans from a psychological perspective. However, these studies have not always properly accounted for the sensitivity of LLMs' behavior to hyperparameters and variations in the prompt. In this study, we examine LLMs' performance on the Horizon decision making task studied by Binz and Schulz (2023) analyzing how LLMs respond to variations in prompts and hyperparameters. By experimenting on three OpenAI language models possessing different capabilities, we observe that the decision making abilities fluctuate based on the input prompts and temperature settings. Contrary to previous findings language models display a human-like exploration exploitation tradeoff after simple adjustments to the prompt. △ Less

Submitted 29 December, 2023; originally announced December 2023.

Comments: EMNLP 2023

Fair Notification Optimization: An Auction Approach

Authors: Christian Kroer, Deeksha Sinha, Xuan Zhang, Shiwen Cheng, Ziyu Zhou

Abstract: Notifications are important for the user experience in mobile apps and can influence their engagement. However, too many notifications can be disruptive for users. A typical mobile app usually has several types of notification, managed by distinct teams with objectives that are possibly conflicting with each other, or even with the overall platform objective. Therefore, there is a need for careful… ▽ More Notifications are important for the user experience in mobile apps and can influence their engagement. However, too many notifications can be disruptive for users. A typical mobile app usually has several types of notification, managed by distinct teams with objectives that are possibly conflicting with each other, or even with the overall platform objective. Therefore, there is a need for careful curation of notifications sent to users of these different types. In this work, we study a novel centralized approach for notification optimization, where we view the opportunities to send user notifications as items and types of notifications as buyers in an auction market. Furthermore, the auction setup is unique, and the platform has the ability to subsidize the bids from the notification types. Using tools from fair division, we study the application of competitive equilibrium for addressing this problem. We show that an Eisenberg-Gale-style convex program allows us to find an allocation that is fair to all notification types in hindsight. Using the dual of the formulation, we present an online algorithm that allocates notifications via first-price auctions using a pacing-multiplier approach. Secondly, we introduce an approach based on second-price auctions and pacing, which has the benefit of working well with existing advertising systems built for second-price auctions. Through an A/B test in production, we show that the second price-based auction system improves over a decentralized notification optimization system, leading to its launch in production for some Instagram notifications. Further, through simulations on Instagram notification data and a subsequent production A/B test, we compare the outcomes of first-price and second-price auctions and show that the former has more stable pacing multipliers. △ Less

Submitted 9 February, 2023; originally announced February 2023.

Bandits for Online Calibration: An Application to Content Moderation on Social Media Platforms

Authors: Vashist Avadhanula, Omar Abdul Baki, Hamsa Bastani, Osbert Bastani, Caner Gocmen, Daniel Haimovich, Darren Hwang, Dima Karamshuk, Thomas Leeper, Jiayuan Ma, Gregory Macnamara, Jake Mullett, Christopher Palow, Sung Park, Varun S Rajagopal, Kevin Schaeffer, Parikshit Shah, Deeksha Sinha, Nicolas Stier-Moses, Peng Xu

Abstract: We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation t… ▽ More We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation trends change over time, affecting which risk models are most reliable. Our system additionally handles production challenges such as changing risk models and novel risk models. We use a contextual bandit to update the calibration in response to such trends. Our approach increases Meta's top-line metric for measuring the effectiveness of its content moderation strategy by 13%. △ Less

Submitted 11 November, 2022; originally announced November 2022.

Symmetry Perception by Deep Networks: Inadequacy of Feed-Forward Architectures and Improvements with Recurrent Connections

Authors: Shobhita Sundaram, Darius Sinha, Matthew Groth, Tomotake Sasaki, Xavier Boix

Abstract: Symmetry is omnipresent in nature and perceived by the visual system of many species, as it facilitates detecting ecologically important classes of objects in our environment. Symmetry perception requires abstraction of long-range spatial dependencies between image regions, and its underlying neural mechanisms remain elusive. In this paper, we evaluate Deep Neural Network (DNN) architectures on th… ▽ More Symmetry is omnipresent in nature and perceived by the visual system of many species, as it facilitates detecting ecologically important classes of objects in our environment. Symmetry perception requires abstraction of long-range spatial dependencies between image regions, and its underlying neural mechanisms remain elusive. In this paper, we evaluate Deep Neural Network (DNN) architectures on the task of learning symmetry perception from examples. We demonstrate that feed-forward DNNs that excel at modelling human performance on object recognition tasks, are unable to acquire a general notion of symmetry. This is the case even when the DNNs are architected to capture long-range spatial dependencies, such as through `dilated' convolutions and the recently introduced `transformers' design. By contrast, we find that recurrent architectures are capable of learning to perceive symmetry by decomposing the long-range spatial dependencies into a sequence of local operations, that are reusable for novel images. These results suggest that recurrent connections likely play an important role in symmetry perception in artificial systems, and possibly, biological ones too. △ Less

Submitted 21 January, 2022; v1 submitted 8 December, 2021; originally announced December 2021.

Pandemic and disability: Challenges faced and role of technology

Authors: Monnie Parida, Dr Manjira Sinha

Abstract: The pandemic has affected every facet of human life. Apart from individuals psychological and mental health issues, the concern regarding mobility, access and communication with high risk infection is a challenging situation. People with disability are more likely vulnerable to infections. The new changes in our social lifestyle (social distancing, limiting touch) can profoundly impact the day tod… ▽ More The pandemic has affected every facet of human life. Apart from individuals psychological and mental health issues, the concern regarding mobility, access and communication with high risk infection is a challenging situation. People with disability are more likely vulnerable to infections. The new changes in our social lifestyle (social distancing, limiting touch) can profoundly impact the day today life of people with disability. In this paper, we will briefly discuss the situation faced by individuals with disabilities, some known remedies, and yet to be identified and curated technological remedies; the impact due to transition of special education toward online mode. Tips and tricks for better utilization of work from home concept by people with disabilities. Accessibility must be universal, accommodating all and encouraging inclusivity. As rightly said by Helen Keller, 'The only thing worse than being blind is having sight but no vision'; subsequently, going by the demand of the time, we should contribute toward the universal design approach by supporting people with disabilities and commit to the changes required in disability care to reduce the impact of pandemic. Keywords: Disabilities, pandemic, corona virus, inclusive △ Less

Submitted 5 August, 2021; v1 submitted 3 August, 2021; originally announced August 2021.

Comments: 10 pages. Paper accepted in IOS Press (Technology and Disability)

Optimizing Offer Sets in Sub-Linear Time

Authors: Vivek F. Farias, Andrew A. Li, Deeksha Sinha

Abstract: Personalization and recommendations are now accepted as core competencies in just about every online setting, ranging from media platforms to e-commerce to social networks. While the challenge of estimating user preferences has garnered significant attention, the operational problem of using such preferences to construct personalized offer sets to users is still a challenge, particularly in modern… ▽ More Personalization and recommendations are now accepted as core competencies in just about every online setting, ranging from media platforms to e-commerce to social networks. While the challenge of estimating user preferences has garnered significant attention, the operational problem of using such preferences to construct personalized offer sets to users is still a challenge, particularly in modern settings where a massive number of items and a millisecond response time requirement mean that even enumerating all of the items is impossible. Faced with such settings, existing techniques are either (a) entirely heuristic with no principled justification, or (b) theoretically sound, but simply too slow to work. Thus motivated, we propose an algorithm for personalized offer set optimization that runs in time sub-linear in the number of items while enjoying a uniform performance guarantee. Our algorithm works for an extremely general class of problems and models of user choice that includes the mixed multinomial logit model as a special case. We achieve a sub-linear runtime by leveraging the dimensionality reduction from learning an accurate latent factor model, along with existing sub-linear time approximate near neighbor algorithms. Our algorithm can be entirely data-driven, relying on samples of the user, where a `sample' refers to the user interaction data typically collected by firms. We evaluate our approach on a massive content discovery dataset from Outbrain that includes millions of advertisements. Results show that our implementation indeed runs fast and with increased performance relative to existing fast heuristics. △ Less

Submitted 17 November, 2020; originally announced November 2020.

Comments: 30 pages, 3 figures. Proceedings of the 21st ACM Conference on Economics and Computation. 2020

ACM Class: G.2.1; F.2.0; E.1; H.3.3

Multi-armed Bandits with Cost Subsidy

Authors: Deeksha Sinha, Karthik Abinav Sankararama, Abbas Kazerouni, Vashist Avadhanula

Abstract: In this paper, we consider a novel variant of the multi-armed bandit (MAB) problem, MAB with cost subsidy, which models many real-life applications where the learning agent has to pay to select an arm and is concerned about optimizing cumulative costs and rewards. We present two applications, intelligent SMS routing problem and ad audience optimization problem faced by several businesses (especial… ▽ More In this paper, we consider a novel variant of the multi-armed bandit (MAB) problem, MAB with cost subsidy, which models many real-life applications where the learning agent has to pay to select an arm and is concerned about optimizing cumulative costs and rewards. We present two applications, intelligent SMS routing problem and ad audience optimization problem faced by several businesses (especially online platforms), and show how our problem uniquely captures key features of these applications. We show that naive generalizations of existing MAB algorithms like Upper Confidence Bound and Thompson Sampling do not perform well for this problem. We then establish a fundamental lower bound on the performance of any online learning algorithm for this problem, highlighting the hardness of our problem in comparison to the classical MAB problem. We also present a simple variant of explore-then-commit and establish near-optimal regret bounds for this algorithm. Lastly, we perform extensive numerical simulations to understand the behavior of a suite of algorithms for various instances and recommend a practical guide to employ different algorithms. △ Less

Submitted 15 March, 2021; v1 submitted 3 November, 2020; originally announced November 2020.

Linear discriminant initialization for feed-forward neural networks

Authors: Marissa Masden, Dev Sinha

Abstract: Informed by the basic geometry underlying feed forward neural networks, we initialize the weights of the first layer of a neural network using the linear discriminants which best distinguish individual classes. Networks initialized in this way take fewer training steps to reach the same level of training, and asymptotically have higher accuracy on training data. Informed by the basic geometry underlying feed forward neural networks, we initialize the weights of the first layer of a neural network using the linear discriminants which best distinguish individual classes. Networks initialized in this way take fewer training steps to reach the same level of training, and asymptotically have higher accuracy on training data. △ Less

Submitted 18 August, 2020; v1 submitted 24 July, 2020; originally announced July 2020.

Comments: 6 pages, 7 figures. Added comparison to larger randomly initialized networks. Updated tables to better illustrate trends in effect size

Multi-Purchase Behavior: Modeling, Estimation and Optimization

Authors: Theja Tulabandhula, Deeksha Sinha, Saketh Reddy Karra, Prasoon Patidar

Abstract: We study the problem of modeling purchase of multiple products and utilizing it to display optimized recommendations for online retailers and e-commerce platforms. We present a parsimonious multi-purchase family of choice models called the Bundle-MVL-K family, and develop a binary search based iterative strategy that efficiently computes optimized recommendations for this model. We establish the… ▽ More We study the problem of modeling purchase of multiple products and utilizing it to display optimized recommendations for online retailers and e-commerce platforms. We present a parsimonious multi-purchase family of choice models called the Bundle-MVL-K family, and develop a binary search based iterative strategy that efficiently computes optimized recommendations for this model. We establish the hardness of computing optimal recommendation sets, and derive several structural properties of the optimal solution that aid in speeding up computation. This is one of the first attempts at operationalizing multi-purchase class of choice models. We show one of the first quantitative links between modeling multiple purchase behavior and revenue gains. The efficacy of our modeling and optimization techniques compared to competing solutions is shown using several real world datasets on multiple metrics such as model fitness, expected revenue gains and run-time reductions. For example, the expected revenue benefit of taking multiple purchases into account is observed to be $\sim5\%$ in relative terms for the Ta Feng and UCI shopping datasets, when compared to the MNL model for instances with $\sim 1500$ products. Additionally, across $6$ real world datasets, the test log-likelihood fits of our models are on average $17\%$ better in relative terms. Our work contributes to the study multi-purchase decisions, analyzing consumer demand and the retailers optimization problem. The simplicity of our models and the iterative nature of our optimization technique allows practitioners meet stringent computational constraints while increasing their revenues in practical recommendation applications at scale, especially in e-commerce platforms and other marketplaces. △ Less

Submitted 5 August, 2023; v1 submitted 14 June, 2020; originally announced June 2020.

Comments: 48 pages. Published in Manufacturing & Service Operations Management 2023

The Limits to Learning a Diffusion Model

Authors: Jackie Baek, Vivek F. Farias, Andreea Georgescu, Retsef Levi, Tianyi Peng, Deeksha Sinha, Joshua Wilde, Andrew Zheng

Abstract: This paper provides the first sample complexity lower bounds for the estimation of simple diffusion models, including the Bass model (used in modeling consumer adoption) and the SIR model (used in modeling epidemics). We show that one cannot hope to learn such models until quite late in the diffusion. Specifically, we show that the time required to collect a number of observations that exceeds our… ▽ More This paper provides the first sample complexity lower bounds for the estimation of simple diffusion models, including the Bass model (used in modeling consumer adoption) and the SIR model (used in modeling epidemics). We show that one cannot hope to learn such models until quite late in the diffusion. Specifically, we show that the time required to collect a number of observations that exceeds our sample complexity lower bounds is large. For Bass models with low innovation rates, our results imply that one cannot hope to predict the eventual number of adopting customers until one is at least two-thirds of the way to the time at which the rate of new adopters is at its peak. In a similar vein, our results imply that in the case of an SIR model, one cannot hope to predict the eventual number of infections until one is approximately two-thirds of the way to the time at which the infection rate has peaked. This lower bound in estimation further translates into a lower bound in regret for decision-making in epidemic interventions. Our results formalize the challenge of accurate forecasting and highlight the importance of incorporating additional data sources. To this end, we analyze the benefit of a seroprevalence study in an epidemic, where we characterize the size of the study needed to improve SIR model estimation. Extensive empirical analyses on product adoption and epidemic data support our theoretical findings. △ Less

Submitted 23 May, 2023; v1 submitted 11 June, 2020; originally announced June 2020.

Optimizing Revenue while showing Relevant Assortments at Scale

Authors: Theja Tulabandhula, Deeksha Sinha, Saketh Karra

Abstract: Scalable real-time assortment optimization has become essential in e-commerce operations due to the need for personalization and the availability of a large variety of items. While this can be done when there are simplistic assortment choices to be made, the optimization process becomes difficult when imposing constraints on the collection of relevant assortments based on insights by store-manager… ▽ More Scalable real-time assortment optimization has become essential in e-commerce operations due to the need for personalization and the availability of a large variety of items. While this can be done when there are simplistic assortment choices to be made, the optimization process becomes difficult when imposing constraints on the collection of relevant assortments based on insights by store-managers and historically well-performing assortments. We design fast and flexible algorithms based on variations of binary search that find the (approximately) optimal assortment in this difficult regime. In particular, we revisit the problem of large-scale assortment optimization under the multinomial logit choice model without any assumptions on the structure of the feasible assortments. We speed up the comparison steps using advances in similarity search in the field of information retrieval/machine learning. For an arbitrary collection of assortments, our algorithms can find a solution in time that is sub-linear in the number of assortments, and for the simpler case of cardinality constraints - linear in the number of items (existing methods are quadratic or worse). Empirical validations using a real world dataset (in addition to experiments using semi-synthetic data based on the Billion Prices dataset and several retail transaction datasets) show that our algorithms are competitive even when the number of items is $\sim 10^5$ ($10\times$ larger instances than previously studied). △ Less

Submitted 1 March, 2021; v1 submitted 6 March, 2020; originally announced March 2020.

Comments: 53 pages, 10 figures

Optimal Scheduling for Maximizing Information Freshness & System Performance in Industrial Cyber-Physical Systems

Authors: Devarpita Sinha, Rajarshi Roy

Abstract: Age of Information is a newly introduced metric, getting vivid attention for measuring the freshness of information in real-time networks. This parameter has evolved to guarantee the reception of timely information from the latest status update, received by a user from any real-time application. In this paper, we study a centralized, closed-loop, networked controlled industrial wireless sensor-act… ▽ More Age of Information is a newly introduced metric, getting vivid attention for measuring the freshness of information in real-time networks. This parameter has evolved to guarantee the reception of timely information from the latest status update, received by a user from any real-time application. In this paper, we study a centralized, closed-loop, networked controlled industrial wireless sensor-actuator network for cyber-physical production systems. Here, we jointly address the problem of transmission scheduling of sensor updates and the restoration of an information flow-line after any real-time update having hard-deadline drops from it, resulting a break in the loop. Unlike existing real-time scheduling policies that only ensure timely updates, this work aims to accomplish both the time-sensitivity and data freshness in new and regenerative real-time updates in terms of the age of information. Here, the coexistence of both cyber and physical units and their individual requirements for providing the quality of service to the system, as a whole, seems to be one of the major challenges to handle. In this work, minimization of staleness of the time-critical updates to extract maximum utilization out of its information content and its effects on other network performances are thoroughly investigated. A greedy scheduling policy called Deadline-aware highest latency first has been used to solve this problem; its performance optimality is proved analytically. Finally, our claim is validated by comparing the results obtained by our algorithm with those of other popular scheduling policies through extensive simulations. △ Less

Submitted 14 January, 2020; v1 submitted 13 January, 2020; originally announced January 2020.

Comments: 21 Pages, 6 figures, 2 tables

Journal ref: Computer Communications: available online 21 January, 2021

Deadline-aware Scheduling for Maximizing Information Freshness in Industrial Cyber-Physical System

Authors: Devarpita Sinha, Rajarshi Roy

Abstract: Age of Information is an interesting metric that captures the freshness of information in the underlying applications. It is a combination of both packets inter-arrival time and packet transmission delay. In recent times, advanced real-time systems rely on this metric for delivering status updates as timely as possible. This paper aims to accomplish optimal transmission scheduling policy to mainta… ▽ More Age of Information is an interesting metric that captures the freshness of information in the underlying applications. It is a combination of both packets inter-arrival time and packet transmission delay. In recent times, advanced real-time systems rely on this metric for delivering status updates as timely as possible. This paper aims to accomplish optimal transmission scheduling policy to maintain the information freshness of real-time updates in the industrial cyber-physical systems. Here the coexistence of both cyber and physical units and their individual requirements to provide the quality of service is one of the critical challenges to handle. A greedy scheduling policy called deadline-aware highest latency first has been proposed for this purpose. This paper also gives the analytical proof of its optimality, and finally, the claim is validated by comparing the performance of our algorithm with other scheduling policies by extensive simulations. △ Less

Submitted 28 September, 2020; v1 submitted 24 December, 2019; originally announced December 2019.

Comments: Accepted in IEEE Sensors Journal

Optimizing Revenue over Data-driven Assortments

Authors: Deeksha Sinha, Theja Tulabandhula

Abstract: We revisit the problem of large-scale assortment optimization under the multinomial logit choice model without any assumptions on the structure of the feasible assortments. Scalable real-time assortment optimization has become essential in e-commerce operations due to the need for personalization and the availability of a large variety of items. While this can be done when there are simplistic ass… ▽ More We revisit the problem of large-scale assortment optimization under the multinomial logit choice model without any assumptions on the structure of the feasible assortments. Scalable real-time assortment optimization has become essential in e-commerce operations due to the need for personalization and the availability of a large variety of items. While this can be done when there are simplistic assortment choices to be made, not imposing any constraints on the collection of feasible assortments gives more flexibility to incorporate insights of store-managers and historically well-performing assortments. We design fast and flexible algorithms based on variations of binary search that find the revenue of the (approximately) optimal assortment. We speed up the comparisons steps using novel vector space embeddings, based on advances in the information retrieval literature. For an arbitrary collection of assortments, our algorithms can find a solution in time that is sub-linear in the number of assortments and for the simpler case of cardinality constraints - linear in the number of items (existing methods are quadratic or worse). Empirical validations using the Billion Prices dataset and several retail transaction datasets show that our algorithms are competitive even when the number of items is $\sim 10^5$ ($100$x larger instances than previously studied). △ Less

Submitted 1 May, 2018; v1 submitted 18 August, 2017; originally announced August 2017.

Comments: 28 pages, 4 figures

On Optimizing Human-Machine Task Assignments

Authors: Andreas Veit, Michael Wilber, Rajan Vaish, Serge Belongie, James Davis, Vishal Anand, Anshu Aviral, Prithvijit Chakrabarty, Yash Chandak, Sidharth Chaturvedi, Chinmaya Devaraj, Ankit Dhall, Utkarsh Dwivedi, Sanket Gupte, Sharath N. Sridhar, Karthik Paga, Anuj Pahuja, Aditya Raisinghani, Ayush Sharma, Shweta Sharma, Darpana Sinha, Nisarg Thakkar, K. Bala Vignesh, Utkarsh Verma, Kanniganti Abhishek , et al. (26 additional authors not shown)

Abstract: When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease… ▽ More When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance. △ Less

Submitted 24 September, 2015; originally announced September 2015.

Comments: HCOMP 2015 Work in Progress