Showing 1–50 of 68 results for author: Sarma, A

Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-informed Neural Networks Framework for Interface Problems

Authors: Sumanta Roy, Chandrasekhar Annavarapu, Pratanu Roy, Antareep Kumar Sarma

Abstract: We present an efficient physics-informed neural networks (PINNs) framework, termed Adaptive Interface-PINNs (AdaI-PINNs), to improve the modeling of interface problems with discontinuous coefficients and/or interfacial jumps. This framework is an enhanced version of its predecessor, Interface PINNs or I-PINNs (Sarma et al.; https://dx.doi.org/10.2139/ssrn.4766623), which involves domain decomposit… ▽ More We present an efficient physics-informed neural networks (PINNs) framework, termed Adaptive Interface-PINNs (AdaI-PINNs), to improve the modeling of interface problems with discontinuous coefficients and/or interfacial jumps. This framework is an enhanced version of its predecessor, Interface PINNs or I-PINNs (Sarma et al.; https://dx.doi.org/10.2139/ssrn.4766623), which involves domain decomposition and assignment of different predefined activation functions to the neural networks in each subdomain across a sharp interface, while keeping all other parameters of the neural networks identical. In AdaI-PINNs, the activation functions vary solely in their slopes, which are trained along with the other parameters of the neural networks. This makes the AdaI-PINNs framework fully automated without requiring preset activation functions. Comparative studies on one-dimensional, two-dimensional, and three-dimensional benchmark elliptic interface problems reveal that AdaI-PINNs outperform I-PINNs, reducing computational costs by 2-6 times while producing similar or better accuracy. △ Less

Submitted 10 June, 2024; v1 submitted 7 June, 2024; originally announced June 2024.

Comments: 17 pages, 8 figures, 6 tables

How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering

Authors: Rudrajit Choudhuri, Dylan Liu, Igor Steinmacher, Marco Gerosa, Anita Sarma

Abstract: Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a con… ▽ More Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Comments: 13 pages, ICSE 2024

Journal ref: ACM/IEEE 46th International Conference on Software Engineering (ICSE 2024)

Make It Make Sense! Understanding and Facilitating Sensemaking in Computational Notebooks

Authors: Souti Chattopadhyay, Zixuan Feng, Emily Arteaga, Audrey Au, Gonzalo Ramos, Titus Barik, Anita Sarma

Abstract: Reusing and making sense of other scientists' computational notebooks. However, making sense of existing notebooks is a struggle, as these reference notebooks are often exploratory, have messy structures, include multiple alternatives, and have little explanation. To help mitigate these issues, we developed a catalog of cognitive tasks associated with the sensemaking process. Utilizing this catalo… ▽ More Reusing and making sense of other scientists' computational notebooks. However, making sense of existing notebooks is a struggle, as these reference notebooks are often exploratory, have messy structures, include multiple alternatives, and have little explanation. To help mitigate these issues, we developed a catalog of cognitive tasks associated with the sensemaking process. Utilizing this catalog, we introduce Porpoise: an interactive overlay on computational notebooks. Porpoise integrates computational notebook features with digital design, grouping cells into labeled sections that can be expanded, collapsed, or annotated for improved sensemaking. We investigated data scientists' needs with unfamiliar computational notebooks and investigated the impact of Porpoise adaptations on their comprehension process. Our counterbalanced study with 24 data scientists found Porpoise enhanced code comprehension, making the experience more akin to reading a book, with one participant describing it as It's really like reading a book. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Comments: 26 Pages

Unveiling Diversity: Empowering OSS Project Leaders with Community Diversity and Turnover Dashboards

Authors: Mariam Guizani, Zixuan Feng, Emily Judith Arteaga, Luis Cañas-Díaz, Alexander Serebrenik, Anita Sarma

Abstract: Managing open-source software (OSS) projects requires managing communities of contributors. In particular, it is essential for project leaders to understand their community's diversity and turnover. We present CommunityTapestry, a dynamic real-time community dashboard, which presents key diversity and turnover signals that we identified from the literature and through participatory design sessions… ▽ More Managing open-source software (OSS) projects requires managing communities of contributors. In particular, it is essential for project leaders to understand their community's diversity and turnover. We present CommunityTapestry, a dynamic real-time community dashboard, which presents key diversity and turnover signals that we identified from the literature and through participatory design sessions with stakeholders. We evaluated CommunityTapestry with an OSS project's contributors and Project Management Committee members, who explored the dashboard using their own project data. Our study results demonstrate that CommunityTapestry increased participants' awareness of their community composition and the diversity and turnover rates in the project. It helped them identify areas of improvement and gave them actionable information. △ Less

Submitted 13 December, 2023; originally announced December 2023.

Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?

Authors: Marco A. Gerosa, Bianca Trinkenreich, Igor Steinmacher, Anita Sarma

Abstract: Research within sociotechnical domains, such as Software Engineering, fundamentally requires a thorough consideration of the human perspective. However, traditional qualitative data collection methods suffer from challenges related to scale, labor intensity, and the increasing difficulty of participant recruitment. This vision paper proposes a novel approach to qualitative data collection in softw… ▽ More Research within sociotechnical domains, such as Software Engineering, fundamentally requires a thorough consideration of the human perspective. However, traditional qualitative data collection methods suffer from challenges related to scale, labor intensity, and the increasing difficulty of participant recruitment. This vision paper proposes a novel approach to qualitative data collection in software engineering research by harnessing the capabilities of artificial intelligence (AI), especially large language models (LLMs) like ChatGPT. We explore the potential of AI-generated synthetic text as an alternative source of qualitative data, by discussing how LLMs can replicate human responses and behaviors in research settings. We examine the application of AI in automating data collection across various methodologies, including persona-based prompting for interviews, multi-persona dialogue for focus groups, and mega-persona responses for surveys. Additionally, we discuss the prospective development of new foundation models aimed at emulating human behavior in observational studies and user evaluations. By simulating human interaction and feedback, these AI models could offer scalable and efficient means of data generation, while providing insights into human attitudes, experiences, and performance. We discuss several open problems and research opportunities to implement this vision and conclude that while AI could augment aspects of data gathering in software engineering research, it cannot replace the nuanced, empathetic understanding inherent in human subjects in some cases, and an integrated approach where both AI and human-generated data coexist will likely yield the most effective outcomes. △ Less

Submitted 18 November, 2023; originally announced November 2023.

SocioEconomicMag Meets a Platform for SES-Diverse College Students: A Case Study

Authors: Puja Agarwal, Divya Prem, Christopher Bogart, Abrar Fallatah, Aileen Abril Castro-Guzman, Pannapat Chanpaisaeng, Stella Doehring, Margaret Burnett, Anita Sarma

Abstract: Emerging research shows that individual differences in how people use technology sometimes cluster by socioeconomic status (SES) and that when technology is not socioeconomically inclusive, low-SES individuals may abandon it. To understand how to improve technology's SES-inclusivity, we present a multi-phase case study on SocioEconomicMag (SESMag), an emerging inspection method for socio+economic… ▽ More Emerging research shows that individual differences in how people use technology sometimes cluster by socioeconomic status (SES) and that when technology is not socioeconomically inclusive, low-SES individuals may abandon it. To understand how to improve technology's SES-inclusivity, we present a multi-phase case study on SocioEconomicMag (SESMag), an emerging inspection method for socio+economic inclusivity. In our 16-month case study, a software team developing a learning management platform used SESMag to evaluate and then to improve their platform's SES-inclusivity. The results showed that (1) the practitioners identified SES-inclusivity bugs in 76% of the features they evaluated; (2) these inclusivity bugs actually arise among low-SES college students; and (3) the SESMag process pointed ways towards fixing these bugs. Finally, (4) a user study with SES-diverse college students showed that the platform's SES-inclusivity eradicated 45-54% of the bugs; for some types of bugs, the bug instance eradication rate was 80% or higher. △ Less

Submitted 10 April, 2023; originally announced April 2023.

Comments: 26 pages, 7 figures

Tag that issue: Applying API-domain labels in issue tracking systems

Authors: Fabio Santos, Joseph Vargovich, Bianca Trinkenreich, Italo Santos, Jacob Penney, Ricardo Britto, João Felipe Pimentel, Igor Wiese, Igor Steinmacher, Anita Sarma, Marco A. Gerosa

Abstract: Labeling issues with the skills required to complete them can help contributors to choose tasks in Open Source Software projects. However, manually labeling issues is time-consuming and error-prone, and current automated approaches are mostly limited to classifying issues as bugs/non-bugs. We investigate the feasibility and relevance of automatically labeling issues with what we call "API-domains,… ▽ More Labeling issues with the skills required to complete them can help contributors to choose tasks in Open Source Software projects. However, manually labeling issues is time-consuming and error-prone, and current automated approaches are mostly limited to classifying issues as bugs/non-bugs. We investigate the feasibility and relevance of automatically labeling issues with what we call "API-domains," which are high-level categories of APIs. Therefore, we posit that the APIs used in the source code affected by an issue can be a proxy for the type of skills (e.g., DB, security, UI) needed to work on the issue. We ran a user study (n=74) to assess API-domain labels' relevancy to potential contributors, leveraged the issues' descriptions and the project history to build prediction models, and validated the predictions with contributors (n=20) of the projects. Our results show that (i) newcomers to the project consider API-domain labels useful in choosing tasks, (ii) labels can be predicted with a precision of 84% and a recall of 78.6% on average, (iii) the results of the predictions reached up to 71.3% in precision and 52.5% in recall when training with a project and testing in another (transfer learning), and (iv) project contributors consider most of the predictions helpful in identifying needed skills. These findings suggest our approach can be applied in practice to automatically label issues, assisting developers in finding tasks that better match their skills. △ Less

Submitted 6 April, 2023; originally announced April 2023.

Comments: arXiv admin note: text overlap with arXiv:2103.12653

ACM Class: D.2.0; I.2.7; K.6.3

Journal ref: Empirical Software Engineering (EMSE), 2023

The State of Diversity and Inclusion in Apache: A Pulse Check

Authors: Zixuan Feng, Mariam Guizani, Marco A. Gerosa, Anita Sarma

Abstract: Diversity and inclusion in open source software (OSS) is a multifaceted concept that arises from differences in contributors' gender, seniority, language, region, and other characteristics. D&I has received growing attention in OSS ecosystems and projects, and various programs have been implemented to foster contributor diversity. However, we do not yet know how the state of D&I is evolving. By un… ▽ More Diversity and inclusion in open source software (OSS) is a multifaceted concept that arises from differences in contributors' gender, seniority, language, region, and other characteristics. D&I has received growing attention in OSS ecosystems and projects, and various programs have been implemented to foster contributor diversity. However, we do not yet know how the state of D&I is evolving. By understanding the state of D&I in OSS projects, the community can develop new and adjust current strategies to foster diversity among contributors and gain insights into the mechanisms and processes that facilitate the development of inclusive communities. In this paper, we report and compare the results of two surveys of Apache Software Foundation (ASF) contributors conducted over two years (n=624 & n=432), considering a variety of D&I aspects. We see improvements in engagement among those traditionally underrepresented in OSS, particularly those who are in gender minority or not confident in English. Yet, the gender gap in the number of contributors remains. We expect this study to help communities tailor their efforts in promoting D&I in OSS. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Comments: 11 pages, 1 figure

Journal ref: 16th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE 2023)

Rules of Engagement: Why and How Companies Participate in OSS

Authors: Mariam Guizani, Aileen Abril Castro-Guzman, Anita Sarma, Igor Steinmacher

Abstract: Company engagement in open source (OSS) is now the new norm. From large technology companies to startups, companies are participating in the OSS ecosystem by open-sourcing their technology, sponsoring projects through funding or paid developer time. However, our understanding of the OSS ecosystem is rooted in the 'old world' model where individual contributors sustain OSS projects. In this work, w… ▽ More Company engagement in open source (OSS) is now the new norm. From large technology companies to startups, companies are participating in the OSS ecosystem by open-sourcing their technology, sponsoring projects through funding or paid developer time. However, our understanding of the OSS ecosystem is rooted in the 'old world' model where individual contributors sustain OSS projects. In this work, we create a more comprehensive understanding of the hybrid OSS landscape by investigating what motivates companies to contribute and how they contribute to OSS. We conducted interviews with 20 participants who have different roles (e.g., CEO, OSPO Lead, Ecosystem Strategist) at 17 different companies of different sizes from large companies (e.g. Microsoft, RedHat, Google, Spotify) to startups. Data from semi-structured interviews reveal that company motivations can be categorized into four levels (Founders' Vision, Reputation, Business Advantage, and Reciprocity) and companies participate through different mechanisms (e.g., Developers' Time, Mentoring Time, Advocacy & Promotion Time), each of which tie to the different types of motivations. We hope our findings nudge more companies to participate in the OSS ecosystem, helping make it robust, diverse, and sustainable. △ Less

Submitted 14 March, 2023; originally announced March 2023.

Comments: 13 pages, 2 figures, ICSE 2023

Designing for Cognitive Diversity: Improving the GitHub Experience for Newcomers

Authors: Italo Santos, João Felipe Pimentel, Igor Wiese, Igor Steinmacher, Anita Sarma, Marco A. Gerosa

Abstract: Social coding platforms such as GitHub have become defacto environments for collaborative programming and open source. When these platforms do not support specific cognitive styles, they create barriers to programming for some populations. Research shows that the cognitive styles typically favored by women are often unsupported, creating barriers to entry for woman newcomers. In this paper, we use… ▽ More Social coding platforms such as GitHub have become defacto environments for collaborative programming and open source. When these platforms do not support specific cognitive styles, they create barriers to programming for some populations. Research shows that the cognitive styles typically favored by women are often unsupported, creating barriers to entry for woman newcomers. In this paper, we use the GenderMag method to evaluate GitHub to find cognitive style-specific inclusivity bugs. We redesigned the "buggy" GitHub features through a web browser plugin, which we evaluated through a between-subjects experiment (n=75). Our results indicate that the changes to the interface improve users' performance and self-efficacy, mainly for individuals with cognitive styles more common to women. Our results can inspire designers of social coding platforms and software engineering tools to produce more inclusive development environments. △ Less

Submitted 10 February, 2023; v1 submitted 25 January, 2023; originally announced January 2023.

Journal ref: 45th IEEE/ACM International Conference on Software Engineering, Software Engineering in Society (ICSE SEIS 2023)

A Model for Understanding and Reducing Developer Burnout

Authors: Bianca Trinkenreich, Klaas-Jan Stol, Igor Steinmacher, Marco Gerosa, Anita Sarma, Marcelo Lara, Michael Feathers, Nicholas Ross, Kevin Bishop

Abstract: Job burnout is a type of work-related stress associated with a state of physical or emotional exhaustion that also involves a sense of reduced accomplishment and loss of personal identity. Burnt out can affect one's physical and mental health and has become a leading industry concern and can result in high workforce turnover. Through an empirical study at Globant, a large multi-national company, w… ▽ More Job burnout is a type of work-related stress associated with a state of physical or emotional exhaustion that also involves a sense of reduced accomplishment and loss of personal identity. Burnt out can affect one's physical and mental health and has become a leading industry concern and can result in high workforce turnover. Through an empirical study at Globant, a large multi-national company, we created a theoretical model to evaluate the complex interplay among organizational culture, work satisfaction, and team climate, and how they impact developer burnout. We conducted a survey of developers in software delivery teams (n=3281) to test our model and analyzed the data using structural equation modeling, moderation, and multi-group analysis. Our results show that Organizational Culture, Climate for Learning, Sense of Belonging, and Inclusiveness are positively associated with Work Satisfaction, which in turn is associated with Reduced Burnout. Our model generated through a large-scale survey can guide organizations in how to reduce workforce burnout by creating a climate for learning, inclusiveness in teams, and a generative organizational culture where new ideas are welcome, information is actively sought and bad news can be shared without fear. △ Less

Submitted 24 January, 2023; v1 submitted 22 January, 2023; originally announced January 2023.

Journal ref: 45th IEEE/ACM International Conference on Software Engineering, Software Engineering in Practice (ICSE SEIP 2023)

Do I Belong? Modeling Sense of Virtual Community Among Linux Kernel Contributors

Authors: Bianca Trinkenreich, Klaas-Jan Stol, Anita Sarma, Daniel M. German, Marco A. Gerosa, Igor Steinmacher

Abstract: The sense of belonging to a community is a basic human need that impacts an individuals behavior, long-term engagement, and job satisfaction, as revealed by research in disciplines such as psychology, healthcare, and education. Despite much research on how to retain developers in Open Source Software projects and other virtual, peer-production communities, there is a paucity of research investigat… ▽ More The sense of belonging to a community is a basic human need that impacts an individuals behavior, long-term engagement, and job satisfaction, as revealed by research in disciplines such as psychology, healthcare, and education. Despite much research on how to retain developers in Open Source Software projects and other virtual, peer-production communities, there is a paucity of research investigating what might contribute to a sense of belonging in these communities. To that end, we develop a theoretical model that seeks to understand the link between OSS developer motives and a Sense of Virtual Community. We test the model with a dataset collected in the Linux Kernel developer community, using structural equation modeling techniques. Our results for this case study show that intrinsic motivations - social or hedonic motives - are positively associated with a sense of virtual community, but living in an authoritative country and being paid to contribute can reduce the sense of virtual community. Based on these results, we offer suggestions for open source projects to foster a sense of virtual community, with a view to retaining contributors and improving projects sustainability. △ Less

Submitted 22 February, 2023; v1 submitted 16 January, 2023; originally announced January 2023.

Journal ref: 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023)

Machine Learning as an Accurate Predictor for Percolation Threshold of Diverse Networks

Authors: Siddharth Patwardhan, Utso Majumder, Aditya Das Sarma, Mayukha Pal, Divyanshi Dwivedi, Prasanta K. Panigrahi

Abstract: The percolation threshold is an important measure to determine the inherent rigidity of large networks. Predictors of the percolation threshold for large networks are computationally intense to run, hence it is a necessity to develop predictors of the percolation threshold of networks, that do not rely on numerical simulations. We demonstrate the efficacy of five machine learning-based regression… ▽ More The percolation threshold is an important measure to determine the inherent rigidity of large networks. Predictors of the percolation threshold for large networks are computationally intense to run, hence it is a necessity to develop predictors of the percolation threshold of networks, that do not rely on numerical simulations. We demonstrate the efficacy of five machine learning-based regression techniques for the accurate prediction of the percolation threshold. The dataset generated to train the machine learning models contains a total of 777 real and synthetic networks. It consists of 5 statistical and structural properties of networks as features and the numerically computed percolation threshold as the output attribute. We establish that the machine learning models outperform three existing empirical estimators of bond percolation threshold, and extend this experiment to predict site and explosive percolation. Further, we compared the performance of our models in predicting the percolation threshold using RMSE values. The gradient boosting regressor, multilayer perceptron and random forests regression models achieve the least RMSE values among considered models. △ Less

Submitted 25 March, 2023; v1 submitted 16 December, 2022; originally announced December 2022.

NFSlicer: Data Movement Optimization for Shallow Network Functions

Authors: Anirudh Sarma, Hamed Seyedroudbari, Harshit Gupta, Umakishore Ramachandran, Alexandros Daglis

Abstract: Network Function (NF) deployments on commodity servers have become ubiquitous in datacenters and enterprise settings. Many commonly used NFs such as firewalls, load balancers and NATs are shallow - i.e., they only examine the packet's header, despite the entire packet being transferred on and off the server. As a result, the gap between moved and inspected data when handling large packets exceeds… ▽ More Network Function (NF) deployments on commodity servers have become ubiquitous in datacenters and enterprise settings. Many commonly used NFs such as firewalls, load balancers and NATs are shallow - i.e., they only examine the packet's header, despite the entire packet being transferred on and off the server. As a result, the gap between moved and inspected data when handling large packets exceeds 20x. At modern network rates, such excess data movement is detrimental to performance, hurting both the average and 90% tail latency of large packets by up to 1.7x. Our thorough performance analysis identifies high contention on the NIC-server PCIe interface and in the server's memory hierarchy as the main bottlenecks. We introduce NFSlicer, a data movement optimization implemented as a NIC extension to mitigate the bottlenecks stemming from data movement deluge in deployments of shallow NFs on commodity servers. NFSlicer only transfers the small portion of each packet that the deployed NFs actually inspect, by slicing the packet's payload and temporarily storing it in on-NIC memory. When the server later transmits the processed packet, NFSlicer splices it to its previously sliced payload. We develop a software-based emulation platform and demonstrate that NFSlicer effectively minimizes data movement between the NIC and the server, bridging the latency gap between small and large packet NF processing. On a range of shallow NFs handling 1518B packets, NFSlicer reduces average and 90% tail latency by up to 17% / 29%, respectively. △ Less

Submitted 4 March, 2022; originally announced March 2022.

Comments: 13 pages, 16 figures

Random Quantum Neural Networks (RQNN) for Noisy Image Recognition

Authors: Debanjan Konar, Erol Gelenbe, Soham Bhandary, Aditya Das Sarma, Attila Cangi

Abstract: Classical Random Neural Networks (RNNs) have demonstrated effective applications in decision making, signal processing, and image recognition tasks. However, their implementation has been limited to deterministic digital systems that output probability distributions in lieu of stochastic behaviors of random spiking signals. We introduce the novel class of supervised Random Quantum Neural Networks… ▽ More Classical Random Neural Networks (RNNs) have demonstrated effective applications in decision making, signal processing, and image recognition tasks. However, their implementation has been limited to deterministic digital systems that output probability distributions in lieu of stochastic behaviors of random spiking signals. We introduce the novel class of supervised Random Quantum Neural Networks (RQNNs) with a robust training strategy to better exploit the random nature of the spiking RNN. The proposed RQNN employs hybrid classical-quantum algorithms with superposition state and amplitude encoding features, inspired by quantum information theory and the brain's spatial-temporal stochastic spiking property of neuron information encoding. We have extensively validated our proposed RQNN model, relying on hybrid classical-quantum algorithms via the PennyLane Quantum simulator with a limited number of \emph{qubits}. Experiments on the MNIST, FashionMNIST, and KMNIST datasets demonstrate that the proposed RQNN model achieves an average classification accuracy of $94.9\%$. Additionally, the experimental findings illustrate the proposed RQNN's effectiveness and resilience in noisy settings, with enhanced image classification accuracy when compared to the classical counterparts (RNNs), classical Spiking Neural Networks (SNNs), and the classical convolutional neural network (AlexNet). Furthermore, the RQNN can deal with noise, which is useful for various applications, including computer vision in NISQ devices. The PyTorch code (https://github.com/darthsimpus/RQN) is made available on GitHub to reproduce the results reported in this manuscript. △ Less

Submitted 3 March, 2022; originally announced March 2022.

How to Debug Inclusivity Bugs? A Debugging Process with Information Architecture

Authors: Mariam Guizani, Igor Steinmacher, Jillian Emard, Abrar Fallatah, Margaret Burnett, Anita Sarma

Abstract: Although some previous research has found ways to find inclusivity bugs (biases in software that introduce inequities), little attention has been paid to how to go about fixing such bugs. Without a process to move from finding to fixing, acting upon such findings is an ad-hoc activity, at the mercy of the skills of each individual developer. To address this gap, we created Why/Where/Fix, a systema… ▽ More Although some previous research has found ways to find inclusivity bugs (biases in software that introduce inequities), little attention has been paid to how to go about fixing such bugs. Without a process to move from finding to fixing, acting upon such findings is an ad-hoc activity, at the mercy of the skills of each individual developer. To address this gap, we created Why/Where/Fix, a systematic inclusivity debugging process whose inclusivity fault localization harnesses Information Architecture(IA) -- the way user-facing information is organized, structured and labeled. We then conducted a multi-stage qualitative empirical evaluation of the effectiveness of Why/Where/Fix, using an Open Source Software (OSS) project's infrastructure as our setting. In our study, the OSS project team used the Why/Where/Fix process to find inclusivity bugs, localize the IA faults behind them, and then fix the IA to remove the inclusivity bugs they had found. Our results showed that using Why/Where/Fix reduced the number of inclusivity bugs that OSS newcomer participants experienced by 90%. △ Less

Submitted 27 February, 2022; originally announced February 2022.

Comments: 12 pages, Accepted at ICSE SEIS 2022

Perceptions of the State of D&I and D&I Initiative in the ASF

Authors: Mariam Guizani, Bianca Trinkenreich, Aileen Abril Castro-Guzman, Igor Steinmacher, Marco Gerosa, Anita Sarma

Abstract: Open Source Software (OSS) Foundations and projects are investing in creating Diversity and Inclusion (D&I) initiatives. However, little is known about contributors' perceptions about the usefulness and success of such initiatives. We aim to close this gap by investigating how contributors perceive the state of D&I in their community. In collaboration with the Apache Software Foundation (ASF), we… ▽ More Open Source Software (OSS) Foundations and projects are investing in creating Diversity and Inclusion (D&I) initiatives. However, little is known about contributors' perceptions about the usefulness and success of such initiatives. We aim to close this gap by investigating how contributors perceive the state of D&I in their community. In collaboration with the Apache Software Foundation (ASF), we surveyed 600+ OSS contributors and conducted 11 follow-up interviews. We used mixed methods to analyze our data-quantitative analysis of Likert-scale questions and qualitative analysis of open-ended survey question and the interviews to understand contributors' perceptions and critiques of the D&I initiative and how to improve it. Our results indicate that the ASF contributors felt that the state of D&I was still lacking, especially regarding gender, seniority, and English proficiency. Regarding the D&I initiative, some participants felt that the effort was unnecessary, while others agreed with the effort but critiqued its implementation. These findings show that D&I initiatives in OSS communities are a good start, but there is room for improvements. Our results can inspire the creation of new and the refinement of current initiatives. △ Less

Submitted 26 February, 2022; originally announced February 2022.

Comments: 13 pages, Accepted at ICSE SEIS 2022

Journal ref: ACM/IEEE 44rd International Conference on Software Engineering, SEIS - Software Engineering in Society (ICSE SEIS 2022)

Implicit Mentoring: The Unacknowledged Developer Efforts in Open Source

Authors: Zixuan Feng, Amreeta Chatterjee, Anita Sarma, Iftekhar Ahmed

Abstract: Mentoring is traditionally viewed as a dyadic, top-down apprenticeship. This perspective, however, overlooks other forms of informal mentoring taking place in everyday activities in which developers invest time and effort, but remain unacknowledged. Here, we investigate the different flavors of mentoring in Open Source Software (OSS) to define and identify implicit mentoring. We first define impli… ▽ More Mentoring is traditionally viewed as a dyadic, top-down apprenticeship. This perspective, however, overlooks other forms of informal mentoring taking place in everyday activities in which developers invest time and effort, but remain unacknowledged. Here, we investigate the different flavors of mentoring in Open Source Software (OSS) to define and identify implicit mentoring. We first define implicit mentoring--situations where contributors guide others through instructions and suggestions embedded in everyday (OSS) activities--through formative interviews with OSS contributors, a literature review, and member-checking. Next, through an empirical investigation of Pull Requests (PRs) in 37 Apache Projects, we build a classifier to extract implicit mentoring and characterize it through the dual lenses of experience and gender. Our analysis of 107,895 PRs shows that implicit mentoring occurs (27.41% of all PRs include implicit mentoring) and it does not follow the traditional dyadic, top-down apprenticeship model. When considering the gender of mentor-mentee pairs, we found pervasive homophily--a preference to mentor those who are of the same gender--in 93.81% cases. In the cross-gender mentoring instances, women were more likely to mentor men. △ Less

Submitted 3 February, 2023; v1 submitted 22 February, 2022; originally announced February 2022.

Comments: 12 pages, 3 figures

Attracting and Retaining OSS Contributors with a Maintainer Dashboard

Authors: Mariam Guizani, Thomas Zimmermann, Anita Sarma, Denae Ford

Abstract: Tools and artifacts produced by open source software (OSS) have been woven into the foundation of the technology industry. To keep this foundation intact, the open source community needs to actively invest in sustainable approaches to bring in new contributors and nurture existing ones. We take a first step at this by collaboratively designing a maintainer dashboard that provides recommendations o… ▽ More Tools and artifacts produced by open source software (OSS) have been woven into the foundation of the technology industry. To keep this foundation intact, the open source community needs to actively invest in sustainable approaches to bring in new contributors and nurture existing ones. We take a first step at this by collaboratively designing a maintainer dashboard that provides recommendations on how to attract and retain open source contributors. For example, by highlighting project goals (e.g., a social good cause) to attract diverse contributors and mechanisms to acknowledge (e.g., a "rising contributor" badge) existing contributors. Next, we conduct a project-specific evaluation with maintainers to better understand use cases in which this tool will be most helpful at supporting their plans for growth. From analyzing feedback, we find recommendations to be useful at signaling projects as welcoming and providing gentle nudges for maintainers to proactively recognize emerging contributors. However, there are complexities to consider when designing recommendations such as the project current development state (e.g., deadlines, milestones, refactoring) and governance model. Finally, we distill our findings to share what the future of recommendations in open source looks like and how to make these recommendations most meaningful over time. △ Less

Submitted 15 February, 2022; originally announced February 2022.

Comments: 5 pages, Accepted at ICSE SEIS 2022

Intersectionality Goes Analytical: Taming Combinatorial Explosion Through Type Abstraction

Authors: Margaret Burnett, Martin Erwig, Abrar Fallatah, Christopher Bogart, Anita Sarma

Abstract: HCI researchers' and practitioners' awareness of intersectionality has been expanding, producing knowledge, recommendations, and prototypes for supporting intersectional populations. However, doing intersectional HCI work is uniquely expensive: it leads to a combinatorial explosion of empirical work (expense 1), and little of the work on one intersectional population can be leveraged to serve anot… ▽ More HCI researchers' and practitioners' awareness of intersectionality has been expanding, producing knowledge, recommendations, and prototypes for supporting intersectional populations. However, doing intersectional HCI work is uniquely expensive: it leads to a combinatorial explosion of empirical work (expense 1), and little of the work on one intersectional population can be leveraged to serve another (expense 2). In this paper, we explain how representations employed by certain analytical design methods correspond to type abstractions, and use that correspondence to identify a (de)compositional model in which a population's diverse identity properties can be joined and split. We formally prove the model's correctness, and show how it enables HCI designers to harness existing analytical HCI methods for use on new intersectional populations of interest. We illustrate through four design use-cases, how the model can reduce the amount of expense 1 and enable designers to leverage prior work to new intersectional populations, addressing expense 2. △ Less

Submitted 25 January, 2022; originally announced January 2022.

A survey on product codes and 2-D codes

Authors: Amajit Sarma

Abstract: One of the simplest way of combining codes to form new codes is to take their direct product. Direct product of cyclic codes and various generalizations have been studied for many years. In this note, we survey cyclic product codes, direct product of various generalizations of cyclic codes and their properties. One of the simplest way of combining codes to form new codes is to take their direct product. Direct product of cyclic codes and various generalizations have been studied for many years. In this note, we survey cyclic product codes, direct product of various generalizations of cyclic codes and their properties. △ Less

Submitted 31 December, 2021; v1 submitted 28 December, 2021; originally announced December 2021.

Comments: 40 pages, typos corrected, references added. arXiv admin note: text overlap with arXiv:1512.06690, arXiv:1505.02238, arXiv:1301.6231 by other authors

Exploiting Activation based Gradient Output Sparsity to Accelerate Backpropagation in CNNs

Authors: Anup Sarma, Sonali Singh, Huaipan Jiang, Ashutosh Pattnaik, Asit K Mishra, Vijaykrishnan Narayanan, Mahmut T Kandemir, Chita R Das

Abstract: Machine/deep-learning (ML/DL) based techniques are emerging as a driving force behind many cutting-edge technologies, achieving high accuracy on computer vision workloads such as image classification and object detection. However, training these models involving large parameters is both time-consuming and energy-hogging. In this regard, several prior works have advocated for sparsity to speed up t… ▽ More Machine/deep-learning (ML/DL) based techniques are emerging as a driving force behind many cutting-edge technologies, achieving high accuracy on computer vision workloads such as image classification and object detection. However, training these models involving large parameters is both time-consuming and energy-hogging. In this regard, several prior works have advocated for sparsity to speed up the of DL training and more so, the inference phase. This work begins with the observation that during training, sparsity in the forward and backward passes are correlated. In that context, we investigate two types of sparsity (input and output type) inherent in gradient descent-based optimization algorithms and propose a hardware micro-architecture to leverage the same. Our experimental results use five state-of-the-art CNN models on the Imagenet dataset, and show back propagation speedups in the range of 1.69$\times$ to 5.43$\times$, compared to the dense baseline execution. By exploiting sparsity in both the forward and backward passes, speedup improvements range from 1.68$\times$ to 3.30$\times$ over the sparsity-agnostic baseline execution. Our work also achieves significant reduction in training iteration time over several previously proposed dense as well as sparse accelerator based platforms, in addition to achieving order of magnitude energy efficiency improvements over GPU based execution. △ Less

Submitted 16 September, 2021; originally announced September 2021.

An Automated Approach to Reasoning About Task-Oriented Insights in Responsive Visualization

Authors: Hyeok Kim, Ryan Rossi, Abhraneel Sarma, Dominik Moritz, Jessica Hullman

Abstract: Authors often transform a large screen visualization for smaller displays through rescaling, aggregation and other techniques when creating visualizations for both desktop and mobile devices (i.e., responsive visualization). However, transformations can alter relationships or patterns implied by the large screen view, requiring authors to reason carefully about what information to preserve while a… ▽ More Authors often transform a large screen visualization for smaller displays through rescaling, aggregation and other techniques when creating visualizations for both desktop and mobile devices (i.e., responsive visualization). However, transformations can alter relationships or patterns implied by the large screen view, requiring authors to reason carefully about what information to preserve while adjusting their design for the smaller display. We propose an automated approach to approximating the loss of support for task-oriented visualization insights (identification, comparison, and trend) in responsive transformation of a source visualization. We operationalize identification, comparison, and trend loss as objective functions calculated by comparing properties of the rendered source visualization to each realized target (small screen) visualization. To evaluate the utility of our approach, we train machine learning models on human ranked small screen alternative visualizations across a set of source visualizations. We find that our approach achieves an accuracy of 84% (random forest model) in ranking visualizations. We demonstrate this approach in a prototype responsive visualization recommender that enumerates responsive transformations using Answer Set Programming and evaluates the preservation of task-oriented insights using our loss measures. We discuss implications of our approach for the development of automated and semi-automated responsive visualization recommendation. △ Less

Submitted 16 July, 2021; originally announced July 2021.

Comments: 9 pages, 11 figures, 2 tables. Accepted at IEEE VIS 2021

Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training

Authors: Anup Sarma, Sonali Singh, Huaipan Jiang, Rui Zhang, Mahmut T Kandemir, Chita R Das

Abstract: Recurrent Neural Networks (RNNs), more specifically their Long Short-Term Memory (LSTM) variants, have been widely used as a deep learning tool for tackling sequence-based learning tasks in text and speech. Training of such LSTM applications is computationally intensive due to the recurrent nature of hidden state computation that repeats for each time step. While sparsity in Deep Neural Nets has b… ▽ More Recurrent Neural Networks (RNNs), more specifically their Long Short-Term Memory (LSTM) variants, have been widely used as a deep learning tool for tackling sequence-based learning tasks in text and speech. Training of such LSTM applications is computationally intensive due to the recurrent nature of hidden state computation that repeats for each time step. While sparsity in Deep Neural Nets has been widely seen as an opportunity for reducing computation time in both training and inference phases, the usage of non-ReLU activation in LSTM RNNs renders the opportunities for such dynamic sparsity associated with neuron activation and gradient values to be limited or non-existent. In this work, we identify dropout induced sparsity for LSTMs as a suitable mode of computation reduction. Dropout is a widely used regularization mechanism, which randomly drops computed neuron values during each iteration of training. We propose to structure dropout patterns, by dropping out the same set of physical neurons within a batch, resulting in column (row) level hidden state sparsity, which are well amenable to computation reduction at run-time in general-purpose SIMD hardware as well as systolic arrays. We conduct our experiments for three representative NLP tasks: language modelling on the PTB dataset, OpenNMT based machine translation using the IWSLT De-En and En-Vi datasets, and named entity recognition sequence labelling using the CoNLL-2003 shared task. We demonstrate that our proposed approach can be used to translate dropout-based computation reduction into reduced training time, with improvement ranging from 1.23x to 1.64x, without sacrificing the target metric. △ Less

Submitted 22 June, 2021; originally announced June 2021.

Pots of Gold at the End of the Rainbow: What is Success for Open Source Contributors?

Authors: Bianca Trinkenreich, Mariam Guizani, Igor Wiese, Tayana Conte, Marco Gerosa, Anita Sarma, Igor Steinmacher

Abstract: Success in Open Source Software (OSS) is often perceived as an exclusively code-centric endeavor. This perception can exclude a variety of individuals with a diverse set of skills and backgrounds, in turn helping create the current diversity & inclusion imbalance in OSS. Because people's perspectives of success affect their personal, professional, and life choices, to be able to support a diverse… ▽ More Success in Open Source Software (OSS) is often perceived as an exclusively code-centric endeavor. This perception can exclude a variety of individuals with a diverse set of skills and backgrounds, in turn helping create the current diversity & inclusion imbalance in OSS. Because people's perspectives of success affect their personal, professional, and life choices, to be able to support a diverse class of individuals, we must first understand what OSS contributors consider successful. Thus far, research has used a uni-dimensional, code-centric lens to define success. In this paper, we challenge this status-quo and reveal the multi-faceted definition of success among OSS contributors. We do so through interviews with 27 OSS contributors who are recognized as successful in their communities, and a follow-up open survey with 193 OSS contributors. Our study provides nuanced definitions of success perceptions in OSS, which might help devise strategies to attract and retain a diverse set of contributors, helping them attain their "pots of gold at the end of the rainbow". △ Less

Submitted 20 July, 2021; v1 submitted 18 May, 2021; originally announced May 2021.

Journal ref: IEEE Transactions on Software Engineering (TSE), 2021

Women's Participation in Open Source Software: A Survey of the Literature

Authors: Bianca Trinkenreich, Igor Wiese, Anita Sarma, Marco Gerosa, Igor Steinmacher

Abstract: Participation of women in Open Source Software (OSS) is very unbalanced, despite various efforts to improve diversity. This is concerning not only because women do not get the chance of career and skill developments afforded by OSS, but also because OSS projects suffer from a lack of diversity of thoughts because of a lack of diversity in their projects. Studies that characterize women's participa… ▽ More Participation of women in Open Source Software (OSS) is very unbalanced, despite various efforts to improve diversity. This is concerning not only because women do not get the chance of career and skill developments afforded by OSS, but also because OSS projects suffer from a lack of diversity of thoughts because of a lack of diversity in their projects. Studies that characterize women's participation and investigate how to attract and retain women are spread across multiple fields, including information systems, software engineering, and social science. This paper systematically maps, aggregates, and synthesizes the state-of-the-art on women's participation in Open Source Software. It focuses on women's representation and the demographics of women who contribute to OSS, how they contribute, the acceptance rates of their contributions, their motivations and challenges, and strategies employed by communities to attract and retain women. We identified 51 articles (published between 2005 and 2021) that investigate women's participation in OSS. According to the literature, women represent about 9.8\% of OSS contributors; most of them are recent contributors, 20-37 years old, devote less than 5h/week to OSS, and make both non-code and code contributions. Only 5\% of projects have women as core developers, and women author less than 5\% of pull-requests but have similar or even higher rates of merge acceptance than men. Besides learning new skills and altruism, reciprocity and kinship are motivations especially relevant for women but can leave if they are not compensated for their contributions. Women's challenges are mainly social, including lack of peer parity and non-inclusive communication from a toxic culture. The literature reports ten strategies, which were mapped to six of the seven challenges. Based on these results, we provide guidelines for future research and practice. △ Less

Submitted 5 January, 2022; v1 submitted 18 May, 2021; originally announced May 2021.

Journal ref: ACM Transactions on Software Engineering and Methodology (TOSEM 2022)

Can I Solve It? Identifying APIs Required to Complete OSS Task

Authors: Fabio Santos, Igor Wiese, Bianca Trinkenreich, Igor Steinmacher, Anita Sarma, Marco Gerosa

Abstract: Open Source Software projects add labels to open issues to help contributors choose tasks. However, manually labeling issues is time-consuming and error-prone. Current automatic approaches for creating labels are mostly limited to classifying issues as a bug/non-bug. In this paper, we investigate the feasibility and relevance of labeling issues with the domain of the APIs required to complete the… ▽ More Open Source Software projects add labels to open issues to help contributors choose tasks. However, manually labeling issues is time-consuming and error-prone. Current automatic approaches for creating labels are mostly limited to classifying issues as a bug/non-bug. In this paper, we investigate the feasibility and relevance of labeling issues with the domain of the APIs required to complete the tasks. We leverage the issues' description and the project history to build prediction models, which resulted in precision up to 82% and recall up to 97.8%. We also ran a user study (n=74) to assess these labels' relevancy to potential contributors. The results show that the labels were useful to participants in choosing tasks, and the API-domain labels were selected more often than the existing architecture-based labels. Our results can inspire the creation of tools to automatically label issues, helping developers to find tasks that better match their skills. △ Less

Submitted 23 March, 2021; originally announced March 2021.

Journal ref: Mining Software Repositories Conference (MSR 2021), May 2021

The Shifting Sands of Motivation: Revisiting What Drives Contributors in Open Source

Authors: Marco Gerosa, Igor Wiese, Bianca Trinkenreich, Georg Link, Gregorio Robles, Christoph Treude, Igor Steinmacher, Anita Sarma

Abstract: Open Source Software (OSS) has changed drastically over the last decade, with OSS projects now producing a large ecosystem of popular products, involving industry participation, and providing professional career opportunities. But our field's understanding of what motivates people to contribute to OSS is still fundamentally grounded in studies from the early 2000s. With the changed landscape of OS… ▽ More Open Source Software (OSS) has changed drastically over the last decade, with OSS projects now producing a large ecosystem of popular products, involving industry participation, and providing professional career opportunities. But our field's understanding of what motivates people to contribute to OSS is still fundamentally grounded in studies from the early 2000s. With the changed landscape of OSS, it is very likely that motivations to join OSS have also evolved. Through a survey of 242 OSS contributors, we investigate shifts in motivation from three perspectives: (1) the impact of the new OSS landscape, (2) the impact of individuals' personal growth as they become part of OSS communities, and (3) the impact of differences in individuals' demographics. Our results show that some motivations related to social aspects and reputation increased in frequency and that some intrinsic and internalized motivations, such as learning and intellectual stimulation, are still highly relevant. We also found that contributing to OSS often transforms extrinsic motivations to intrinsic, and that while experienced contributors often shift toward altruism, novices often shift toward career, fun, kinship, and learning. OSS projects can leverage our results to revisit current strategies to attract and retain contributors, and researchers and tool builders can better support the design of new studies and tools to engage and support OSS development. △ Less

Submitted 29 January, 2021; v1 submitted 25 January, 2021; originally announced January 2021.

Comments: Paper accept at 43rd ACM/IEEE International Conference on Software Engineering, Madrid, Spain. June 2021

Journal ref: ACM/IEEE 43rd International Conference on Software Engineering (ICSE 2021)

If This Context Then That Concern: Exploring users' concerns with IFTTT applets

Authors: Mahsa Saeidi, McKenzie Calvert, Audrey W. Au, Anita Sarma, Rakesh B. Bobba

Abstract: End users are increasingly using trigger-action platforms like, If-This-Then-That (IFTTT) to create applets to connect smart home devices and services. However, there are inherent risks in using such applets -- even non-malicious ones -- as sensitive information may leak through their use in certain contexts (e.g., where the device is located, who can observe the resultant action). This work aims… ▽ More End users are increasingly using trigger-action platforms like, If-This-Then-That (IFTTT) to create applets to connect smart home devices and services. However, there are inherent risks in using such applets -- even non-malicious ones -- as sensitive information may leak through their use in certain contexts (e.g., where the device is located, who can observe the resultant action). This work aims to understand how well end users can assess this risk. We do so by exploring users' concerns with using IFTTT applets and more importantly if and how those concerns change based on different contextual factors. Through a Mechanical Turk survey of 386 participants on 49 smart-home IFTTT applets, we found that nudging the participants to think about different usage contexts led them to think deeper about the associated risks and raise their concerns. Qualitative analysis reveals that participants had a nuanced understanding of contextual factors and how these factors could lead to leakage of sensitive data and allow unauthorized access to applets and data. △ Less

Submitted 23 December, 2020; originally announced December 2020.

Replacements and Replaceables: Making the Case for Code Variants

Authors: Venkatesh Vinayakarao, Devika Sondhi, Sumit Keswani, Rahul Purandare, Anita Sarma

Abstract: There are often multiple ways to implement the same requirement in source code. Different implementation choices can result in code snippets that are similar, and have been defined in multiple ways: code clones, examples, simions and variants. Currently, there is a lack of a consistent and unambiguous definition of such types of code snippets. Here we present a characterization study of code varia… ▽ More There are often multiple ways to implement the same requirement in source code. Different implementation choices can result in code snippets that are similar, and have been defined in multiple ways: code clones, examples, simions and variants. Currently, there is a lack of a consistent and unambiguous definition of such types of code snippets. Here we present a characterization study of code variants - a specific type of code snippets that differ from each other by at least one desired property, within a given code context. We distinguish code variants from other types of redundancies in source code, and demonstrate the significant role that they play: about 25% to 43% of developer discussions (in a set of nine open source projects) were about variants. We characterize different types of variants based on their code context and desired properties. As a demonstration of the possible use of our characterization of code variants, we show how search results can be ranked based on a desired property (e.g., speed of execution). △ Less

Submitted 14 June, 2020; v1 submitted 6 June, 2020; originally announced June 2020.

ACM Class: D.2.m; D.2.13

Quantum Unsupervised and Supervised Learning on Superconducting Processors

Authors: Abhijat Sarma, Rupak Chatterjee, Kaitlin Gili, Ting Yu

Abstract: Machine learning algorithms perform well on identifying patterns in many different datasets due to their versatility. However, as one increases the size of the dataset, the computation time for training and using these statistical models grows quickly. Quantum computing offers a new paradigm which may have the ability to overcome these computational difficulties. Here, we propose a quantum analogu… ▽ More Machine learning algorithms perform well on identifying patterns in many different datasets due to their versatility. However, as one increases the size of the dataset, the computation time for training and using these statistical models grows quickly. Quantum computing offers a new paradigm which may have the ability to overcome these computational difficulties. Here, we propose a quantum analogue to K-means clustering, implement it on simulated superconducting qubits, and compare it to a previously developed quantum support vector machine. We find the algorithm's accuracy comparable to the classical K-means algorithm for clustering and classification problems, and find that it has asymptotic complexity $O(N^{3/2}K^{1/2}\log{P})$, where $N$ is the number of data points, $K$ is the number of clusters, and $P$ is the dimension of the data points, giving a significant speedup over the classical analogue. △ Less

Submitted 6 January, 2022; v1 submitted 9 September, 2019; originally announced September 2019.

Comments: Updated to the published version

Journal ref: Quantum Information and Computation 20 (7&8), 541-552 (2020)

Engineering Gender-Inclusivity into Software: Tales from the Trenches

Authors: Claudia Hilderbrand, Christopher Perdriau, Lara Letaw, Jillian Emard, Zoe Steine-Hanson, Margaret Burnett, Anita Sarma

Abstract: Although the need for gender-inclusivity in software itself is gaining attention among both SE researchers and SE practitioners, and methods have been published to help, little has been reported on how to make such methods work in real-world settings. For example, how do busy software practitioners use such methods in low-cost ways? How do they endeavor to maximize benefits from using them? How do… ▽ More Although the need for gender-inclusivity in software itself is gaining attention among both SE researchers and SE practitioners, and methods have been published to help, little has been reported on how to make such methods work in real-world settings. For example, how do busy software practitioners use such methods in low-cost ways? How do they endeavor to maximize benefits from using them? How do they avoid the controversies that can arise in talking about gender? To find out how teams were handling these and similar questions, we turned to 10 real-world software teams. We present these teams experiences "in the trenches," in the form of 12 practices and 3 potential pitfalls, so as to provide their insights to other real-world software teams trying to engineer gender-inclusivity into their software products. △ Less

Submitted 27 May, 2019; originally announced May 2019.

Fixing Inclusivity Bugs for Information Processing Styles and Learning Styles

Authors: Zoe Steine-Hanson, Claudia Hilderbrand, Lara Letaw, Jillian Emard, Christopher Perdriau, Christopher Mendez, Margaret Burnett, Anita Sarma

Abstract: Most software systems today do not support cognitive diversity. Further, because of differences in problem-solving styles that cluster by gender, software that poorly supports cognitive diversity can also embed gender biases. To help software professionals fix gender bias "bugs" related to people's problem-solving styles for information processing and learning of new software we collected inclusiv… ▽ More Most software systems today do not support cognitive diversity. Further, because of differences in problem-solving styles that cluster by gender, software that poorly supports cognitive diversity can also embed gender biases. To help software professionals fix gender bias "bugs" related to people's problem-solving styles for information processing and learning of new software we collected inclusivity fixes from three sources. The first two are empirical studies we conducted: a heuristics-driven user study and a field research industry study. The third is data that we obtained about a before/after user study of inclusivity bugs. The resulting seven potential inclusivity fixes show how to debug software to be more inclusive for diverse problem-solving styles. △ Less

Submitted 7 May, 2019; originally announced May 2019.

From GenderMag to InclusiveMag: An Inclusive Design Meta-Method

Authors: Christopher Mendez, Lara Letaw, Margaret Burnett, Simone Stumpf, Anita Sarma, Claudia Hilderbrand

Abstract: How can software practitioners assess whether their software supports diverse users? Although there are empirical processes that can be used to find "inclusivity bugs" piecemeal, what is often needed is a systematic inspection method to assess soft-ware's support for diverse populations. To help fill this gap, this paper introduces InclusiveMag, a generalization of GenderMag that can be used to ge… ▽ More How can software practitioners assess whether their software supports diverse users? Although there are empirical processes that can be used to find "inclusivity bugs" piecemeal, what is often needed is a systematic inspection method to assess soft-ware's support for diverse populations. To help fill this gap, this paper introduces InclusiveMag, a generalization of GenderMag that can be used to generate systematic inclusiveness methods for a particular dimension of diversity. We then present a multi-case study covering eight diversity dimensions, of eight teams' experiences applying InclusiveMag to eight under-served populations and their "mainstream" counterparts. △ Less

Submitted 7 May, 2019; originally announced May 2019.

EvoCut : A new Generalization of Albert-Barabási Model for Evolution of Complex Networks

Authors: Shailesh Kumar Jaiswal, Nabajyoti Medhi, Manjish Pal, Mridul Sahu, Prashant Sahu, Amal Dev Sarma

Abstract: With the evolution of social networks, the network structure shows dynamic nature in which nodes and edges appear as well as disappear for various reasons. The role of a node in the network is presented as the number of interactions it has with the other nodes. For this purpose a network is modeled as a graph where nodes represent network members and edges represent a relationship among them. Seve… ▽ More With the evolution of social networks, the network structure shows dynamic nature in which nodes and edges appear as well as disappear for various reasons. The role of a node in the network is presented as the number of interactions it has with the other nodes. For this purpose a network is modeled as a graph where nodes represent network members and edges represent a relationship among them. Several models for evolution of social networks has been proposed till date, most widely accepted being the Barabási-Albert \cite{Network science} model that is based on \emph{preferential attachment} of nodes according to the degree distribution. This model leads to generation of graphs that are called \emph{Scale Free} and the degree distribution of such graphs follow the \emph{power law}. Several generalizations of this model has also been proposed. In this paper we present a new generalization of the model and attempt to bring out its implications in real life. △ Less

Submitted 1 March, 2018; originally announced March 2018.

Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace

Authors: Ayush Jain, Akash Das Sarma, Aditya Parameswaran, Jennifer Widom

Abstract: We conduct an experimental analysis of a dataset comprising over 27 million microtasks performed by over 70,000 workers issued to a large crowdsourcing marketplace between 2012-2016. Using this data---never before analyzed in an academic context---we shed light on three crucial aspects of crowdsourcing: (1) Task design --- helping requesters understand what constitutes an effective task, and how t… ▽ More We conduct an experimental analysis of a dataset comprising over 27 million microtasks performed by over 70,000 workers issued to a large crowdsourcing marketplace between 2012-2016. Using this data---never before analyzed in an academic context---we shed light on three crucial aspects of crowdsourcing: (1) Task design --- helping requesters understand what constitutes an effective task, and how to go about designing one; (2) Marketplace dynamics --- helping marketplace administrators and designers understand the interaction between tasks and workers, and the corresponding marketplace load; and (3) Worker behavior --- understanding worker attention spans, lifetimes, and general behavior, for the improvement of the crowdsourcing ecosystem as a whole. △ Less

Submitted 22 January, 2017; originally announced January 2017.

Optimizing Open-Ended Crowdsourcing: The Next Frontier in Crowdsourced Data Management

Authors: Aditya Parameswaran, Akash Das Sarma, Vipul Venkataraman

Abstract: Crowdsourcing is the primary means to generate training data at scale, and when combined with sophisticated machine learning algorithms, crowdsourcing is an enabler for a variety of emergent automated applications impacting all spheres of our lives. This paper surveys the emerging field of formally reasoning about and optimizing open-ended crowdsourcing, a popular and crucially important, but seve… ▽ More Crowdsourcing is the primary means to generate training data at scale, and when combined with sophisticated machine learning algorithms, crowdsourcing is an enabler for a variety of emergent automated applications impacting all spheres of our lives. This paper surveys the emerging field of formally reasoning about and optimizing open-ended crowdsourcing, a popular and crucially important, but severely understudied class of crowdsourcing---the next frontier in crowdsourced data management. The underlying challenges include distilling the right answer when none of the workers agree with each other, teasing apart the various perspectives adopted by workers when answering tasks, and effectively selecting between the many open-ended operators appropriate for a problem. We describe the approaches that we've found to be effective for open-ended crowdsourcing, drawing from our experiences in this space. △ Less

Submitted 17 October, 2016; originally announced October 2016.

Fusing Data with Correlations

Authors: Ravali Pochampally, Anish Das Sarma, Xin Luna Dong, Alexandra Meliou, Divesh Srivastava

Abstract: Many applications rely on Web data and extraction systems to accomplish knowledge-driven tasks. Web information is not curated, so many sources provide inaccurate, or conflicting information. Moreover, extraction systems introduce additional noise to the data. We wish to automatically distinguish correct data and erroneous data for creating a cleaner set of integrated data. Previous work has shown… ▽ More Many applications rely on Web data and extraction systems to accomplish knowledge-driven tasks. Web information is not curated, so many sources provide inaccurate, or conflicting information. Moreover, extraction systems introduce additional noise to the data. We wish to automatically distinguish correct data and erroneous data for creating a cleaner set of integrated data. Previous work has shown that a naïve voting strategy that trusts data provided by the majority or at least a certain number of sources may not work well in the presence of copying between the sources. However, correlation between sources can be much broader than copying: sources may provide data from complementary domains (\emph{negative correlation}), extractors may focus on different types of information (\emph{negative correlation}), and extractors may apply common rules in extraction (\emph{positive correlation, without copying}). In this paper we present novel techniques modeling correlations between sources and applying it in truth finding. △ Less

Submitted 1 March, 2015; originally announced March 2015.

Comments: Sigmod'2014

Globally Optimal Crowdsourcing Quality Management

Authors: Akash Das Sarma, Aditya Parameswaran, Jennifer Widom

Abstract: We study crowdsourcing quality management, that is, given worker responses to a set of tasks, our goal is to jointly estimate the true answers for the tasks, as well as the quality of the workers. Prior work on this problem relies primarily on applying Expectation-Maximization (EM) on the underlying maximum likelihood problem to estimate true answers as well as worker quality. Unfortunately, EM on… ▽ More We study crowdsourcing quality management, that is, given worker responses to a set of tasks, our goal is to jointly estimate the true answers for the tasks, as well as the quality of the workers. Prior work on this problem relies primarily on applying Expectation-Maximization (EM) on the underlying maximum likelihood problem to estimate true answers as well as worker quality. Unfortunately, EM only provides a locally optimal solution rather than a globally optimal one. Other solutions to the problem (that do not leverage EM) fail to provide global optimality guarantees as well. In this paper, we focus on filtering, where tasks require the evaluation of a yes/no predicate, and rating, where tasks elicit integer scores from a finite domain. We design algorithms for finding the global optimal estimates of correct task answers and worker quality for the underlying maximum likelihood problem, and characterize the complexity of these algorithms. Our algorithms conceptually consider all mappings from tasks to true answers (typically a very large number), leveraging two key ideas to reduce, by several orders of magnitude, the number of mappings under consideration, while preserving optimality. We also demonstrate that these algorithms often find more accurate estimates than EM-based algorithms. This paper makes an important contribution towards understanding the inherent complexity of globally optimal crowdsourcing quality management. △ Less

Submitted 1 March, 2015; v1 submitted 26 February, 2015; originally announced February 2015.

Fast Approximate Matching of Cell-Phone Videos for Robust Background Subtraction

Authors: Raffay Hamid, Atish Das Sarma, Dennis DeCoste, Neel Sundaresan

Abstract: We identify a novel instance of the background subtraction problem that focuses on extracting near-field foreground objects captured using handheld cameras. Given two user-generated videos of a scene, one with and the other without the foreground object(s), our goal is to efficiently generate an output video with only the foreground object(s) present in it. We cast this challenge as a spatio-tempo… ▽ More We identify a novel instance of the background subtraction problem that focuses on extracting near-field foreground objects captured using handheld cameras. Given two user-generated videos of a scene, one with and the other without the foreground object(s), our goal is to efficiently generate an output video with only the foreground object(s) present in it. We cast this challenge as a spatio-temporal frame matching problem, and propose an efficient solution for it that exploits the temporal smoothness of the video sequences. We present theoretical analyses for the error bounds of our approach, and validate our findings using a detailed set of simulation experiments. Finally, we present the results of our approach tested on multiple real videos captured using handheld cameras, and compare them to several alternate foreground extraction approaches. △ Less

Submitted 21 April, 2014; originally announced April 2014.

Distributed Computation of Sparse Cuts

Authors: Atish Das Sarma, Anisur Rahaman Molla, Gopal Pandurangan

Abstract: Finding sparse cuts is an important tool in analyzing large-scale distributed networks such as the Internet and Peer-to-Peer networks, as well as large-scale graphs such as the web graph, online social communities, and VLSI circuits. In distributed communication networks, they are useful for topology maintenance and for designing better search and routing algorithms. In this paper, we focus on d… ▽ More Finding sparse cuts is an important tool in analyzing large-scale distributed networks such as the Internet and Peer-to-Peer networks, as well as large-scale graphs such as the web graph, online social communities, and VLSI circuits. In distributed communication networks, they are useful for topology maintenance and for designing better search and routing algorithms. In this paper, we focus on developing fast distributed algorithms for computing sparse cuts in networks. Given an undirected $n$-node network $G$ with conductance $φ$, the goal is to find a cut set whose conductance is close to $φ$. We present two distributed algorithms that find a cut set with sparsity $\tilde O(\sqrtφ)$ ($\tilde{O}$ hides $\polylog{n}$ factors). Both our algorithms work in the CONGEST distributed computing model and output a cut of conductance at most $\tilde O(\sqrtφ)$ with high probability, in $\tilde O(\frac{1}{b}(\frac{1}φ + n))$ rounds, where $b$ is balance of the cut of given conductance. In particular, to find a sparse cut of constant balance, our algorithms take $\tilde O(\frac{1}φ + n)$ rounds. Our algorithms can also be used to output a {\em local} cluster, i.e., a subset of vertices near a given source node, and whose conductance is within a quadratic factor of the best possible cluster around the specified node. Both our distributed algorithm can work without knowledge of the optimal $φ$ value and hence can be used to find approximate conductance values both globally and with respect to a given source node. We also give a lower bound on the time needed for any distributed algorithm to compute any non-trivial sparse cut --- any distributed approximation algorithm (for any non-trivial approximation ratio) for computing sparsest cut will take $\tilde Ω(\sqrt{n} + D)$ rounds, where $D$ is the diameter of the graph. △ Less

Submitted 20 October, 2013; originally announced October 2013.

Comments: 19 pages, 1 figure

Distributed Random Walks

Authors: Atish Das Sarma, Danupon Nanongkai, Gopal Pandurangan, Prasad Tetali

Abstract: Performing random walks in networks is a fundamental primitive that has found applications in many areas of computer science, including distributed computing. In this paper, we focus on the problem of sampling random walks efficiently in a distributed network and its applications. Given bandwidth constraints, the goal is to minimize the number of rounds required to obtain random walk samples. Al… ▽ More Performing random walks in networks is a fundamental primitive that has found applications in many areas of computer science, including distributed computing. In this paper, we focus on the problem of sampling random walks efficiently in a distributed network and its applications. Given bandwidth constraints, the goal is to minimize the number of rounds required to obtain random walk samples. All previous algorithms that compute a random walk sample of length $\ell$ as a subroutine always do so naively, i.e., in $O(\ell)$ rounds. The main contribution of this paper is a fast distributed algorithm for performing random walks. We present a sublinear time distributed algorithm for performing random walks whose time complexity is sublinear in the length of the walk. Our algorithm performs a random walk of length $\ell$ in $\tilde{O}(\sqrt{\ell D})$ rounds ($\tilde{O}$ hides $\polylog{n}$ factors where $n$ is the number of nodes in the network) with high probability on an undirected network, where $D$ is the diameter of the network. For small diameter graphs, this is a significant improvement over the naive $O(\ell)$ bound. Furthermore, our algorithm is optimal within a poly-logarithmic factor as there exists a matching lower bound [Nanongkai et al. PODC 2011]. We further extend our algorithms to efficiently perform $k$ independent random walks in $\tilde{O}(\sqrt{k\ell D} + k)$ rounds. We also show that our algorithm can be applied to speedup the more general Metropolis-Hastings sampling. Our random walk algorithms can be used to speed up distributed algorithms in applications that use random walks as a subroutine, such as computing a random spanning tree and estimating mixing time and related parameters. Our algorithm is fully decentralized and can serve as a building block in the design of topologically-aware networks. △ Less

Submitted 19 February, 2013; originally announced February 2013.

Comments: Preprint of an article to appear in Journal of the ACM in February 2013. The official journal version has several gramatical corrections. Preliminary versions of this paper appeared in PODC 2009 and PODC 2010. arXiv admin note: substantial text overlap with arXiv:0911.3195, arXiv:1205.5525

ACM Class: F.2.2; G.2.2

Fast Distributed PageRank Computation

Authors: Atish Das Sarma, Anisur Rahaman Molla, Gopal Pandurangan, Eli Upfal

Abstract: Over the last decade, PageRank has gained importance in a wide range of applications and domains, ever since it first proved to be effective in determining node importance in large graphs (and was a pioneering idea behind Google's search engine). In distributed computing alone, PageRank vector, or more generally random walk based quantities have been used for several different applications ranging… ▽ More Over the last decade, PageRank has gained importance in a wide range of applications and domains, ever since it first proved to be effective in determining node importance in large graphs (and was a pioneering idea behind Google's search engine). In distributed computing alone, PageRank vector, or more generally random walk based quantities have been used for several different applications ranging from determining important nodes, load balancing, search, and identifying connectivity structures. Surprisingly, however, there has been little work towards designing provably efficient fully-distributed algorithms for computing PageRank. The difficulty is that traditional matrix-vector multiplication style iterative methods may not always adapt well to the distributed setting owing to communication bandwidth restrictions and convergence rates. In this paper, we present fast random walk-based distributed algorithms for computing PageRanks in general graphs and prove strong bounds on the round complexity. We first present a distributed algorithm that takes $O\big(\log n/\eps \big)$ rounds with high probability on any graph (directed or undirected), where $n$ is the network size and $\eps$ is the reset probability used in the PageRank computation (typically $\eps$ is a fixed constant). We then present a faster algorithm that takes $O\big(\sqrt{\log n}/\eps \big)$ rounds in undirected graphs. Both of the above algorithms are scalable, as each node sends only small ($\polylog n$) number of bits over each edge per round. To the best of our knowledge, these are the first fully distributed algorithms for computing PageRank vector with provably efficient running time. △ Less

Submitted 25 November, 2015; v1 submitted 15 August, 2012; originally announced August 2012.

Comments: 14 pages

Journal ref: Theoretical Computer Science, Volume 561, Pages 113-121, 2015

Dense Subgraphs on Dynamic Networks

Authors: Atish Das Sarma, Ashwin Lall, Danupon Nanongkai, Amitabh Trehan

Abstract: In distributed networks, it is often useful for the nodes to be aware of dense subgraphs, e.g., such a dense subgraph could reveal dense subtructures in otherwise sparse graphs (e.g. the World Wide Web or social networks); these might reveal community clusters or dense regions for possibly maintaining good communication infrastructure. In this work, we address the problem of self-awareness of node… ▽ More In distributed networks, it is often useful for the nodes to be aware of dense subgraphs, e.g., such a dense subgraph could reveal dense subtructures in otherwise sparse graphs (e.g. the World Wide Web or social networks); these might reveal community clusters or dense regions for possibly maintaining good communication infrastructure. In this work, we address the problem of self-awareness of nodes in a dynamic network with regards to graph density, i.e., we give distributed algorithms for maintaining dense subgraphs that the member nodes are aware of. The only knowledge that the nodes need is that of the dynamic diameter $D$, i.e., the maximum number of rounds it takes for a message to traverse the dynamic network. For our work, we consider a model where the number of nodes are fixed, but a powerful adversary can add or remove a limited number of edges from the network at each time step. The communication is by broadcast only and follows the CONGEST model. Our algorithms are continuously executed on the network, and at any time (after some initialization) each node will be aware if it is part (or not) of a particular dense subgraph. We give algorithms that ($2 + ε$)-approximate the densest subgraph and ($3 + ε$)-approximate the at-least-$k$-densest subgraph (for a given parameter $k$). Our algorithms work for a wide range of parameter values and run in $O(D\log_{1+ε} n)$ time. Further, a special case of our results also gives the first fully decentralized approximation algorithms for densest and at-least-$k$-densest subgraph problems for static distributed graphs. △ Less

Submitted 7 August, 2012; originally announced August 2012.

Comments: To appear in the 26th International Symposium on Distributed Computing (DISC 2012)

ACM Class: C.2.4; F.0; F.2.2; G.2.2

Upper and Lower Bounds on the Cost of a Map-Reduce Computation

Authors: Foto N. Afrati, Anish Das Sarma, Semih Salihoglu, Jeffrey D. Ullman

Abstract: In this paper we study the tradeoff between parallelism and communication cost in a map-reduce computation. For any problem that is not "embarrassingly parallel," the finer we partition the work of the reducers so that more parallelism can be extracted, the greater will be the total communication between mappers and reducers. We introduce a model of problems that can be solved in a single round of… ▽ More In this paper we study the tradeoff between parallelism and communication cost in a map-reduce computation. For any problem that is not "embarrassingly parallel," the finer we partition the work of the reducers so that more parallelism can be extracted, the greater will be the total communication between mappers and reducers. We introduce a model of problems that can be solved in a single round of map-reduce computation. This model enables a generic recipe for discovering lower bounds on communication cost as a function of the maximum number of inputs that can be assigned to one reducer. We use the model to analyze the tradeoff for three problems: finding pairs of strings at Hamming distance $d$, finding triangles and other patterns in a larger graph, and matrix multiplication. For finding strings of Hamming distance 1, we have upper and lower bounds that match exactly. For triangles and many other graphs, we have upper and lower bounds that are the same to within a constant factor. For the problem of matrix multiplication, we have matching upper and lower bounds for one-round map-reduce algorithms. We are also able to explore two-round map-reduce algorithms for matrix multiplication and show that these never have more communication, for a given reducer size, than the best one-round algorithm, and often have significantly less. △ Less

Submitted 19 June, 2012; originally announced June 2012.

Comments: 14 pages

Fast Distributed Computation in Dynamic Networks via Random Walks

Authors: Atish Das Sarma, Anisur Rahaman Molla, Gopal Pandurangan

Abstract: The paper investigates efficient distributed computation in dynamic networks in which the network topology changes (arbitrarily) from round to round. Our first contribution is a rigorous framework for design and analysis of distributed random walk algorithms in dynamic networks. We then develop a fast distributed random walk based algorithm that runs in $\tilde{O}(\sqrt{τΦ})$ rounds (with high p… ▽ More The paper investigates efficient distributed computation in dynamic networks in which the network topology changes (arbitrarily) from round to round. Our first contribution is a rigorous framework for design and analysis of distributed random walk algorithms in dynamic networks. We then develop a fast distributed random walk based algorithm that runs in $\tilde{O}(\sqrt{τΦ})$ rounds (with high probability), where $τ$ is the dynamic mixing time and $Φ$ is the dynamic diameter of the network respectively, and returns a sample close to a suitably defined stationary distribution of the dynamic network. We also apply our fast random walk algorithm to devise fast distributed algorithms for two key problems, namely, information dissemination and decentralized computation of spectral properties in a dynamic network. Our next contribution is a fast distributed algorithm for the fundamental problem of information dissemination (also called as gossip) in a dynamic network. In gossip, or more generally, $k$-gossip, there are $k$ pieces of information (or tokens) that are initially present in some nodes and the problem is to disseminate the $k$ tokens to all nodes. We present a random-walk based algorithm that runs in $\tilde{O}(\min\{n^{1/3}k^{2/3}(τΦ)^{1/3}, nk\})$ rounds (with high probability). To the best of our knowledge, this is the first $o(nk)$-time fully-distributed token forwarding algorithm that improves over the previous-best $O(nk)$ round distributed algorithm [Kuhn et al., STOC 2010], although in an oblivious adversary model. Our final contribution is a simple and fast distributed algorithm for estimating the dynamic mixing time and related spectral properties of the underlying dynamic network. △ Less

Submitted 24 May, 2012; originally announced May 2012.

Your Two Weeks of Fame and Your Grandmother's

Authors: James Cook, Atish Das Sarma, Alex Fabrikant, Andrew Tomkins

Abstract: Did celebrity last longer in 1929, 1992 or 2009? We investigate the phenomenon of fame by mining a collection of news articles that spans the twentieth century, and also perform a side study on a collection of blog posts from the last 10 years. By analyzing mentions of personal names, we measure each person's time in the spotlight, using two simple metrics that evaluate, roughly, the duration of a… ▽ More Did celebrity last longer in 1929, 1992 or 2009? We investigate the phenomenon of fame by mining a collection of news articles that spans the twentieth century, and also perform a side study on a collection of blog posts from the last 10 years. By analyzing mentions of personal names, we measure each person's time in the spotlight, using two simple metrics that evaluate, roughly, the duration of a single news story about a person, and the overall duration of public interest in a person. We watched the distribution evolve from 1895 to 2010, expecting to find significantly shortening fame durations, per the much popularly bemoaned shortening of society's attention spans and quickening of media's news cycles. Instead, we conclusively demonstrate that, through many decades of rapid technological and societal change, through the appearance of Twitter, communication satellites, and the Internet, fame durations did not decrease, neither for the typical case nor for the extremely famous, with the last statistically significant fame duration decreases coming in the early 20th century, perhaps from the spread of telegraphy and telephony. Furthermore, while median fame durations stayed persistently constant, for the most famous of the famous, as measured by either volume or duration of media attention, fame durations have actually trended gently upward since the 1940s, with statistically significant increases on 40-year timescales. Similar studies have been done with much shorter timescales specifically in the context of information spreading on Twitter and similar social networking sites. To the best of our knowledge, this is the first massive scale study of this nature that spans over a century of archived data, thereby allowing us to track changes across decades. △ Less

Submitted 19 April, 2012; originally announced April 2012.

Comments: This version supercedes the short version of this paper published in the proceedings of WWW 2012

ACM Class: J.4

Vision Paper: Towards an Understanding of the Limits of Map-Reduce Computation

Authors: Foto N. Afrati, Anish Das Sarma, Semih Salihoglu, Jeffrey D. Ullman

Abstract: A significant amount of recent research work has addressed the problem of solving various data management problems in the cloud. The major algorithmic challenges in map-reduce computations involve balancing a multitude of factors such as the number of machines available for mappers/reducers, their memory requirements, and communication cost (total amount of data sent from mappers to reducers). Mos… ▽ More A significant amount of recent research work has addressed the problem of solving various data management problems in the cloud. The major algorithmic challenges in map-reduce computations involve balancing a multitude of factors such as the number of machines available for mappers/reducers, their memory requirements, and communication cost (total amount of data sent from mappers to reducers). Most past work provides custom solutions to specific problems, e.g., performing fuzzy joins in map-reduce, clustering, graph analyses, and so on. While some problems are amenable to very efficient map-reduce algorithms, some other problems do not lend themselves to a natural distribution, and have provable lower bounds. Clearly, the ease of "map-reducability" is closely related to whether the problem can be partitioned into independent pieces, which are distributed across mappers/reducers. What makes a problem distributable? Can we characterize general properties of problems that determine how easy or hard it is to find efficient map-reduce algorithms? This is a vision paper that attempts to answer the questions described above. △ Less

Submitted 8 April, 2012; originally announced April 2012.

Comments: 5 pages

Finding Connected Components on Map-reduce in Logarithmic Rounds

Authors: Vibhor Rastogi, Ashwin Machanavajjhala, Laukik Chitnis, Anish Das Sarma

Abstract: Given a large graph G = (V,E) with millions of nodes and edges, how do we compute its connected components efficiently? Recent work addresses this problem in map-reduce, where a fundamental trade-off exists between the number of map-reduce rounds and the communication of each round. Denoting d the diameter of the graph, and n the number of nodes in the largest component, all prior map-reduce techn… ▽ More Given a large graph G = (V,E) with millions of nodes and edges, how do we compute its connected components efficiently? Recent work addresses this problem in map-reduce, where a fundamental trade-off exists between the number of map-reduce rounds and the communication of each round. Denoting d the diameter of the graph, and n the number of nodes in the largest component, all prior map-reduce techniques either require d rounds, or require about n|V| + |E| communication per round. We propose two randomized map-reduce algorithms -- (i) Hash-Greater-To-Min, which provably requires at most 3log(n) rounds with high probability, and at most 2(|V| + |E|) communication per round, and (ii) Hash-to-Min, which has a worse theoretical complexity, but in practice completes in at most 2log(d) rounds and 3(|V| + |E|) communication per rounds. Our techniques for connected components can be applied to clustering as well. We propose a novel algorithm for agglomerative single linkage clustering in map-reduce. This is the first algorithm that can provably compute a clustering in at most O(log(n)) rounds, where n is the size of the largest cluster. We show the effectiveness of all our algorithms through detailed experiments on large synthetic as well as real-world datasets. △ Less

Submitted 12 November, 2012; v1 submitted 24 March, 2012; originally announced March 2012.

Near-Optimal Random Walk Sampling in Distributed Networks

Authors: Atish Das Sarma, Anisur Rahaman Molla, Gopal Pandurangan

Abstract: Performing random walks in networks is a fundamental primitive that has found numerous applications in communication networks such as token management, load balancing, network topology discovery and construction, search, and peer-to-peer membership management. While several such algorithms are ubiquitous, and use numerous random walk samples, the walks themselves have always been performed naively… ▽ More Performing random walks in networks is a fundamental primitive that has found numerous applications in communication networks such as token management, load balancing, network topology discovery and construction, search, and peer-to-peer membership management. While several such algorithms are ubiquitous, and use numerous random walk samples, the walks themselves have always been performed naively. In this paper, we focus on the problem of performing random walk sampling efficiently in a distributed network. Given bandwidth constraints, the goal is to minimize the number of rounds and messages required to obtain several random walk samples in a continuous online fashion. We present the first round and message optimal distributed algorithms that present a significant improvement on all previous approaches. The theoretical analysis and comprehensive experimental evaluation of our algorithms show that they perform very well in different types of networks of differing topologies. In particular, our results show how several random walks can be performed continuously (when source nodes are provided only at runtime, i.e., online), such that each walk of length $\ell$ can be performed exactly in just $\tilde{O}(\sqrt{\ell D})$ rounds, (where $D$ is the diameter of the network), and $O(\ell)$ messages. This significantly improves upon both, the naive technique that requires $O(\ell)$ rounds and $O(\ell)$ messages, and the sophisticated algorithm of [DasSarma et al. PODC 2010] that has the same round complexity as this paper but requires $Ω(m\sqrt{\ell})$ messages (where $m$ is the number of edges in the network). Our theoretical results are corroborated through extensive experiments on various topological data sets. Our algorithms are fully decentralized, lightweight, and easily implementable, and can serve as building blocks in the design of topologically-aware networks. △ Less

Submitted 11 January, 2012; v1 submitted 6 January, 2012; originally announced January 2012.