-
The Life Cycle of Large Language Models: A Review of Biases in Education
Authors:
Jinsook Lee,
Yann Hicke,
Renzhe Yu,
Christopher Brooks,
René F. Kizilcec
Abstract:
Large Language Models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM-based applications to understand and generate natural language can potentially improve instructional effectiveness and learning outcomes, but the integration of LLMs in education technology has renewed concerns over algorithmic bi…
▽ More
Large Language Models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM-based applications to understand and generate natural language can potentially improve instructional effectiveness and learning outcomes, but the integration of LLMs in education technology has renewed concerns over algorithmic bias which may exacerbate educational inequities. In this review, building on prior work on mapping the traditional machine learning life cycle, we provide a holistic map of the LLM life cycle from the initial development of LLMs to customizing pre-trained models for various applications in educational settings. We explain each step in the LLM life cycle and identify potential sources of bias that may arise in the context of education. We discuss why current measures of bias from traditional machine learning fail to transfer to LLM-generated content in education, such as tutoring conversations because the text is high-dimensional, there can be multiple correct responses, and tailoring responses may be pedagogically desirable rather than unfair. This review aims to clarify the complex nature of bias in LLM applications and provide practical guidance for their evaluation to promote educational equity.
△ Less
Submitted 3 June, 2024;
originally announced July 2024.
-
Fairness Hub Technical Briefs: Definition and Detection of Distribution Shift
Authors:
Nicolas Acevedo,
Carmen Cortez,
Chris Brooks,
Rene Kizilcec,
Renzhe Yu
Abstract:
Distribution shift is a common situation in machine learning tasks, where the data used for training a model is different from the data the model is applied to in the real world. This issue arises across multiple technical settings: from standard prediction tasks, to time-series forecasting, and to more recent applications of large language models (LLMs). This mismatch can lead to performance redu…
▽ More
Distribution shift is a common situation in machine learning tasks, where the data used for training a model is different from the data the model is applied to in the real world. This issue arises across multiple technical settings: from standard prediction tasks, to time-series forecasting, and to more recent applications of large language models (LLMs). This mismatch can lead to performance reductions, and can be related to a multiplicity of factors: sampling issues and non-representative data, changes in the environment or policies, or the emergence of previously unseen scenarios. This brief focuses on the definition and detection of distribution shifts in educational settings. We focus on standard prediction problems, where the task is to learn a model that takes in a series of input (predictors) $X=(x_1,x_2,...,x_m)$ and produces an output $Y=f(X)$.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
"Don't Step on My Toes": Resolving Editing Conflicts in Real-Time Collaboration in Computational Notebooks
Authors:
April Yi Wang,
Zihan Wu,
Christopher Brooks,
Steve Oney
Abstract:
Real-time collaborative editing in computational notebooks can improve the efficiency of teamwork for data scientists. However, working together through synchronous editing of notebooks introduces new challenges. Data scientists may inadvertently interfere with each others' work by altering the shared codebase and runtime state if they do not set up a social protocol for working together and monit…
▽ More
Real-time collaborative editing in computational notebooks can improve the efficiency of teamwork for data scientists. However, working together through synchronous editing of notebooks introduces new challenges. Data scientists may inadvertently interfere with each others' work by altering the shared codebase and runtime state if they do not set up a social protocol for working together and monitoring their collaborators' progress. In this paper, we propose a real-time collaborative editing model for resolving conflict edits in computational notebooks that introduces three levels of edit protection to help collaborators avoid introducing errors to both the program source code and changes to the runtime state.
△ Less
Submitted 6 April, 2024;
originally announced April 2024.
-
Probabilistically-Sound Beam Search with Masked Language Models
Authors:
Creston Brooks,
Robert Calef,
Charlie Cowen-Breen,
Anna Sappington
Abstract:
Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. However, estimating such distributions has important domain-specific applications such as ancient text restoration and protein engineering. Here we present probabilistically-sound methods for beam search with MLMs.…
▽ More
Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. However, estimating such distributions has important domain-specific applications such as ancient text restoration and protein engineering. Here we present probabilistically-sound methods for beam search with MLMs. First, we clarify the conditions under which it is theoretically sound to perform text infilling with MLMs using standard beam search. When these conditions fail, we provide a probabilistically-sound modification with no additional computational complexity and demonstrate that it is superior to the aforementioned beam search in the expected conditions. We then present empirical results comparing several infilling approaches with MLMs across several domains.
△ Less
Submitted 9 July, 2024; v1 submitted 22 February, 2024;
originally announced February 2024.
-
Bridging Learnersourcing and AI: Exploring the Dynamics of Student-AI Collaborative Feedback Generation
Authors:
Anjali Singh,
Christopher Brooks,
Xu Wang,
Warren Li,
Juho Kim,
Deepti Pandey
Abstract:
This paper explores the space of optimizing feedback mechanisms in complex domains, such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment condu…
▽ More
This paper explores the space of optimizing feedback mechanisms in complex domains, such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment conducted with 72 Master's level students in a data visualization course, comparing two conditions: students writing hints independently versus revising hints generated by GPT-4. The study aimed to evaluate the quality of learnersourced hints, examine the impact of student performance on hint quality, gauge learner preference for writing hints with or without AI support, and explore the potential of the student-AI collaborative exercise in fostering critical thinking about LLMs. Based on our findings, we provide insights for designing learnersourcing activities leveraging AI support and optimizing students' learning as they interact with LLMs.
△ Less
Submitted 20 November, 2023;
originally announced November 2023.
-
Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation
Authors:
Tung Phung,
Victor-Alexandru Pădurean,
Anjali Singh,
Christopher Brooks,
José Cambronero,
Sumit Gulwani,
Adish Singla,
Gustavo Soares
Abstract:
Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation…
▽ More
Generative AI and large language models hold great promise in enhancing programming education by automatically generating individualized feedback for students. We investigate the role of generative AI models in providing human tutor-style programming hints to help students resolve errors in their buggy programs. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios; however, their overall quality is still inferior to human tutors and not yet ready for real-world deployment. In this paper, we seek to push the limits of generative AI models toward providing high-quality programming hints and develop a novel technique, GPT4Hints-GPT3.5Val. As a first step, our technique leverages GPT-4 as a ``tutor'' model to generate hints -- it boosts the generative quality by using symbolic information of failing test cases and fixes in prompts. As a next step, our technique leverages GPT-3.5, a weaker model, as a ``student'' model to further validate the hint quality -- it performs an automatic quality validation by simulating the potential utility of providing this feedback. We show the efficacy of our technique via extensive evaluation using three real-world datasets of Python programs covering a variety of concepts ranging from basic algorithms to regular expressions and data analysis using pandas library.
△ Less
Submitted 6 August, 2024; v1 submitted 5 October, 2023;
originally announced October 2023.
-
Fairness Hub Technical Briefs: AUC Gap
Authors:
Jinsook Lee,
Chris Brooks,
Renzhe Yu,
Rene Kizilcec
Abstract:
To measure bias, we encourage teams to consider using AUC Gap: the absolute difference between the highest and lowest test AUC for subgroups (e.g., gender, race, SES, prior knowledge). It is agnostic to the AI/ML algorithm used and it captures the disparity in model performance for any number of subgroups, which enables non-binary fairness assessments such as for intersectional identity groups. Th…
▽ More
To measure bias, we encourage teams to consider using AUC Gap: the absolute difference between the highest and lowest test AUC for subgroups (e.g., gender, race, SES, prior knowledge). It is agnostic to the AI/ML algorithm used and it captures the disparity in model performance for any number of subgroups, which enables non-binary fairness assessments such as for intersectional identity groups. The teams use a wide range of AI/ML models in pursuit of a common goal of doubling math achievement in low-income middle schools. Ensuring that the models, which are trained on datasets collected in many different contexts, do not introduce or amplify biases is important for achieving the goal. We offer here a versatile and easy-to-compute measure of model bias for all the teams in order to create a common benchmark and an analytical basis for sharing what strategies have worked for different teams.
△ Less
Submitted 25 September, 2023; v1 submitted 20 September, 2023;
originally announced September 2023.
-
Logion: Machine Learning for Greek Philology
Authors:
Charlie Cowen-Breen,
Creston Brooks,
Johannes Haubold,
Barbara Graziosi
Abstract:
This paper presents machine-learning methods to address various problems in Greek philology. After training a BERT model on the largest premodern Greek dataset used for this purpose to date, we identify and correct previously undetected errors made by scribes in the process of textual transmission, in what is, to our knowledge, the first successful identification of such errors via machine learnin…
▽ More
This paper presents machine-learning methods to address various problems in Greek philology. After training a BERT model on the largest premodern Greek dataset used for this purpose to date, we identify and correct previously undetected errors made by scribes in the process of textual transmission, in what is, to our knowledge, the first successful identification of such errors via machine learning. Additionally, we demonstrate the model's capacity to fill gaps caused by material deterioration of premodern manuscripts and compare the model's performance to that of a domain expert. We find that best performance is achieved when the domain expert is provided with model suggestions for inspiration. With such human-computer collaborations in mind, we explore the model's interpretability and find that certain attention heads appear to encode select grammatical features of premodern Greek.
△ Less
Submitted 1 May, 2023;
originally announced May 2023.
-
Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity
Authors:
Josh Gardner,
Renzhe Yu,
Quan Nguyen,
Christopher Brooks,
Rene Kizilcec
Abstract:
Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We prop…
▽ More
Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We propose a framework and metrics for assessing the utility and fairness of student dropout prediction models that are transferred across institutions. We examine the feasibility of cross-institutional transfer under real-world data- and model-sharing constraints, quantifying model biases for intersectional student identities, characterizing potential disparate impact due to these biases, and investigating the impact of various cross-institutional ensembling approaches on fairness and overall model performance. We perform this analysis on data representing over 200,000 enrolled students annually from four universities without sharing training data between institutions.
We find that a simple zero-shot cross-institutional transfer procedure can achieve similar performance to locally-trained models for all institutions in our study, without sacrificing model fairness. We also find that stacked ensembling provides no additional benefits to overall performance or fairness compared to either a local model or the zero-shot transfer procedure we tested. We find no evidence of a fairness-accuracy tradeoff across dozens of models and transfer schemes evaluated. Our auditing procedure also highlights the importance of intersectional fairness analysis, revealing performance disparities at the intersection of sensitive identity groups that are concealed under one-dimensional analysis.
△ Less
Submitted 1 May, 2023;
originally announced May 2023.
-
FLiMS: a Fast Lightweight 2-way Merger for Sorting
Authors:
Philippos Papaphilippou,
Wayne Luk,
Chris Brooks
Abstract:
In this paper, we present FLiMS, a highly-efficient and simple parallel algorithm for merging two sorted lists residing in banked and/or wide memory. On FPGAs, its implementation uses fewer hardware resources than the state-of-the-art alternatives, due to the reduced number of comparators and elimination of redundant logic found on prior attempts. In combination with the distributed nature of the…
▽ More
In this paper, we present FLiMS, a highly-efficient and simple parallel algorithm for merging two sorted lists residing in banked and/or wide memory. On FPGAs, its implementation uses fewer hardware resources than the state-of-the-art alternatives, due to the reduced number of comparators and elimination of redundant logic found on prior attempts. In combination with the distributed nature of the selector stage, a higher performance is achieved for the same amount of parallelism or higher. This is useful in many applications such as in parallel merge trees to achieve high-throughput sorting, where the resource utilisation of the merger is critical for building large trees and internalising the workload for fast computation. Also presented are efficient variations of FLiMS for optimizing throughput for skewed datasets, achieving stable sorting or using fewer dequeue signals. Additionally, FLiMS is shown to perform well as conventional software on modern CPUs supporting single-instruction multiple-data (SIMD) instructions, surpassing the performance of some standard libraries for sorting.
△ Less
Submitted 7 March, 2022; v1 submitted 10 December, 2021;
originally announced December 2021.
-
Visualization of Intended Assistance for Acceptance of Shared Control
Authors:
Connor Brooks,
Daniel Szafir
Abstract:
In shared control, advances in autonomous robotics are applied to help empower a human user in operating a robotic system. While these systems have been shown to improve efficiency and operation success, users are not always accepting of the new control paradigm produced by working with an assistive controller. This mismatch between performance and acceptance can prevent users from taking advantag…
▽ More
In shared control, advances in autonomous robotics are applied to help empower a human user in operating a robotic system. While these systems have been shown to improve efficiency and operation success, users are not always accepting of the new control paradigm produced by working with an assistive controller. This mismatch between performance and acceptance can prevent users from taking advantage of the benefits of shared control systems for robotic operation. To address this mismatch, we develop multiple types of visualizations for improving both the legibility and perceived predictability of assistive controllers, then conduct a user study to evaluate the impact that these visualizations have on user acceptance of shared control systems. Our results demonstrate that shared control visualizations must be designed carefully to be effective, with users requiring visualizations that improve both legibility and predictability of the assistive controller in order to voluntarily relinquish control.
△ Less
Submitted 24 August, 2020;
originally announced August 2020.
-
Isometric force pillow: using air pressure to quantify involuntary finger flexion in the presence of hypertonia
Authors:
Caitlyn E. Seim,
Chuzhang Han,
Alexis J. Lowber,
Claire Brooks,
Marie Payne,
Maarten G. Lansberg,
Kara E. Flavin,
Julius P. A. Dewald,
Allison M. Okamura
Abstract:
Survivors of central nervous system injury commonly present with spastic hypertonia. The affected muscles are hyperexcitable and can display involuntary static muscle tone and an exaggerated stretch reflex. These symptoms affect posture and disrupt activities of daily living. Symptoms are typically measured using subjective manual tests such as the Modified Ashworth Scale; however, more quantitati…
▽ More
Survivors of central nervous system injury commonly present with spastic hypertonia. The affected muscles are hyperexcitable and can display involuntary static muscle tone and an exaggerated stretch reflex. These symptoms affect posture and disrupt activities of daily living. Symptoms are typically measured using subjective manual tests such as the Modified Ashworth Scale; however, more quantitative measures are necessary to evaluate potential treatments. The hands are one of the most common targets for intervention, but few investigators attempt to quantify symptoms of spastic hypertonia affecting the fingers. We present the isometric force pillow (IFP) to quantify involuntary grip force. This lightweight, computerized tool provides a holistic measure of finger flexion force and can be used in various orientations for clinical testing and to measure the impact of assistive devices.
△ Less
Submitted 21 August, 2020;
originally announced August 2020.
-
Building Second-Order Mental Models for Human-Robot Interaction
Authors:
Connor Brooks,
Daniel Szafir
Abstract:
The mental models that humans form of other agents---encapsulating human beliefs about agent goals, intentions, capabilities, and more---create an underlying basis for interaction. These mental models have the potential to affect both the human's decision making during the interaction and the human's subjective assessment of the interaction. In this paper, we surveyed existing methods for modeling…
▽ More
The mental models that humans form of other agents---encapsulating human beliefs about agent goals, intentions, capabilities, and more---create an underlying basis for interaction. These mental models have the potential to affect both the human's decision making during the interaction and the human's subjective assessment of the interaction. In this paper, we surveyed existing methods for modeling how humans view robots, then identified a potential method for improving these estimates through inferring a human's model of a robot agent directly from their actions. Then, we conducted an online study to collect data in a grid-world environment involving humans moving an avatar past a virtual agent. Through our analysis, we demonstrated that participants' action choices leaked information about their mental models of a virtual agent. We conclude by discussing the implications of these findings and the potential for such a method to improve human-robot interactions.
△ Less
Submitted 13 September, 2019;
originally announced September 2019.
-
Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments
Authors:
Timothy NeCamp,
Josh Gardner,
Christopher Brooks
Abstract:
Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet l…
▽ More
Randomized experiments ensure robust causal inference that are critical to effective learning analytics research and practice. However, traditional randomized experiments, like A/B tests, are limiting in large scale digital learning environments. While traditional experiments can accurately compare two treatment options, they are less able to inform how to adapt interventions to continually meet learners' diverse needs. In this work, we introduce a trial design for developing adaptive interventions in scaled digital learning environments -- the sequential randomized trial (SRT). With the goal of improving learner experience and developing interventions that benefit all learners at all times, SRTs inform how to sequence, time, and personalize interventions. In this paper, we provide an overview of SRTs, and we illustrate the advantages they hold compared to traditional experiments. We describe a novel SRT run in a large scale data science MOOC. The trial results contextualize how learner engagement can be addressed through inclusive culturally targeted reminder emails. We also provide practical advice for researchers who aim to run their own SRTs to develop adaptive interventions in scaled digital learning environments.
△ Less
Submitted 31 January, 2019; v1 submitted 26 October, 2018;
originally announced October 2018.
-
Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining
Authors:
Josh Gardner,
Yuming Yang,
Ryan Baker,
Christopher Brooks
Abstract:
The use of machine learning techniques has expanded in education research, driven by the rich data from digital learning environments and institutional data warehouses. However, replication of machine learned models in the domain of the learning sciences is particularly challenging due to a confluence of experimental, methodological, and data barriers. We discuss the challenges of end-to-end machi…
▽ More
The use of machine learning techniques has expanded in education research, driven by the rich data from digital learning environments and institutional data warehouses. However, replication of machine learned models in the domain of the learning sciences is particularly challenging due to a confluence of experimental, methodological, and data barriers. We discuss the challenges of end-to-end machine learning replication in this context, and present an open-source software toolkit, the MOOC Replication Framework (MORF), to address them. We demonstrate the use of MORF by conducting a replication at scale, and provide a complete executable container, with unique DOIs documenting the configurations of each individual trial, for replication or future extension at https://github.com/educational-technology-collective/fy2015-replication. This work demonstrates an approach to end-to-end machine learning replication which is relevant to any domain with large, complex or multi-format, privacy-protected data with a consistent schema.
△ Less
Submitted 10 July, 2018; v1 submitted 13 June, 2018;
originally announced June 2018.
-
Dropout Model Evaluation in MOOCs
Authors:
Josh Gardner,
Christopher Brooks
Abstract:
The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this m…
▽ More
The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this method to a series of algorithms and feature sets derived from a large sample of Massive Open Online Courses (MOOCs). While a complete comparison of all potential modeling approaches is beyond the scope of this paper, we show that this approach reveals a large gap in dropout prediction performance between forum-, assignment-, and clickstream-based feature extraction methods, where the latter is significantly better than the former two, which are in turn indistinguishable from one another. This work has methodological implications for evaluating predictive or AI-based models of student success, and practical implications for the design and targeting of at-risk student models and interventions.
△ Less
Submitted 16 February, 2018;
originally announced February 2018.
-
Evaluating Predictive Models of Student Success: Closing the Methodological Gap
Authors:
Josh Gardner,
Christopher Brooks
Abstract:
Model evaluation -- the process of making inferences about the performance of predictive models -- is a critical component of predictive modeling research in learning analytics. We survey the state of the practice with respect to model evaluation in learning analytics, which overwhelmingly uses only naive methods for model evaluation or statistical tests which are not appropriate for predictive mo…
▽ More
Model evaluation -- the process of making inferences about the performance of predictive models -- is a critical component of predictive modeling research in learning analytics. We survey the state of the practice with respect to model evaluation in learning analytics, which overwhelmingly uses only naive methods for model evaluation or statistical tests which are not appropriate for predictive model evaluation. We conduct a critical comparison of both null hypothesis significance testing (NHST) and a preferred Bayesian method for model evaluation. Finally, we apply three methods -- the na{ï}ve average commonly used in learning analytics, NHST, and Bayesian -- to a predictive modeling experiment on a large set of MOOC data. We compare 96 different predictive models, including different feature sets, statistical modeling algorithms, and tuning hyperparameters for each, using this case study to demonstrate the different experimental conclusions these evaluation techniques provide.
△ Less
Submitted 13 June, 2018; v1 submitted 19 January, 2018;
originally announced January 2018.
-
MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data
Authors:
Josh Gardner,
Christopher Brooks,
Juan Miguel L. Andres,
Ryan Baker
Abstract:
Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We presen…
▽ More
Big data repositories from online learning platforms such as Massive Open Online Courses (MOOCs) represent an unprecedented opportunity to advance research on education at scale and impact a global population of learners. To date, such research has been hindered by poor reproducibility and a lack of replication, largely due to three types of barriers: experimental, inferential, and data. We present a novel system for large-scale computational research, the MOOC Replication Framework (MORF), to jointly address these barriers. We discuss MORF's architecture, an open-source platform-as-a-service (PaaS) which includes a simple, flexible software API providing for multiple modes of research (predictive modeling or production rule analysis) integrated with a high-performance computing environment. All experiments conducted on MORF use executable Docker containers which ensure complete reproducibility while allowing for the use of any software or language which can be installed in the linux-based Docker container. Each experimental artifact is assigned a DOI and made publicly available. MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform. We also highlight ways in which MORF represents a solution template to a more general class of problems faced by computational researchers in other domains.
△ Less
Submitted 21 August, 2018; v1 submitted 16 January, 2018;
originally announced January 2018.
-
Student Success Prediction in MOOCs
Authors:
Josh Gardner,
Christopher Brooks
Abstract:
Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a categorization of MOOC research according to the predictors (features), prediction (outcomes), and underlying theoretical mo…
▽ More
Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a categorization of MOOC research according to the predictors (features), prediction (outcomes), and underlying theoretical model. We critically survey work across each category, providing data on the raw data source, feature engineering, statistical model, evaluation method, prediction architecture, and other aspects of these experiments. Such a review is particularly useful given the rapid expansion of predictive modeling research in MOOCs since the emergence of major MOOC platforms in 2012. This survey reveals several key methodological gaps, which include extensive filtering of experimental subpopulations, ineffective student model evaluation, and the use of experimental data which would be unavailable for real-world student success prediction and intervention, which is the ultimate goal of such models. Finally, we highlight opportunities for future research, which include temporal modeling, research bridging predictive and explanatory student models, work which contributes to learning theory, and evaluating long-term learner success in MOOCs.
△ Less
Submitted 10 April, 2018; v1 submitted 16 November, 2017;
originally announced November 2017.
-
Citing and Reading Behaviours in High-Energy Physics. How a Community Stopped Worrying about Journals and Learned to Love Repositories
Authors:
Anne Gentil-Beccot,
Salvatore Mele,
Travis Brooks
Abstract:
Contemporary scholarly discourse follows many alternative routes in addition to the three-century old tradition of publication in peer-reviewed journals. The field of High- Energy Physics (HEP) has explored alternative communication strategies for decades, initially via the mass mailing of paper copies of preliminary manuscripts, then via the inception of the first online repositories and digita…
▽ More
Contemporary scholarly discourse follows many alternative routes in addition to the three-century old tradition of publication in peer-reviewed journals. The field of High- Energy Physics (HEP) has explored alternative communication strategies for decades, initially via the mass mailing of paper copies of preliminary manuscripts, then via the inception of the first online repositories and digital libraries.
This field is uniquely placed to answer recurrent questions raised by the current trends in scholarly communication: is there an advantage for scientists to make their work available through repositories, often in preliminary form? Is there an advantage to publishing in Open Access journals? Do scientists still read journals or do they use digital repositories?
The analysis of citation data demonstrates that free and immediate online dissemination of preprints creates an immense citation advantage in HEP, whereas publication in Open Access journals presents no discernible advantage. In addition, the analysis of clickstreams in the leading digital library of the field shows that HEP scientists seldom read journals, preferring preprints instead.
△ Less
Submitted 25 November, 2009; v1 submitted 30 June, 2009;
originally announced June 2009.
-
Information Resources in High-Energy Physics: Surveying the Present Landscape and Charting the Future Course
Authors:
Anne Gentil-Beccot,
Salvatore Mele,
Annette Holtkamp,
Heath B. O'Connell,
Travis C. Brooks
Abstract:
Access to previous results is of paramount importance in the scientific process. Recent progress in information management focuses on building e-infrastructures for the optimization of the research workflow, through both policy-driven and user-pulled dynamics. For decades, High-Energy Physics (HEP) has pioneered innovative solutions in the field of information management and dissemination. In li…
▽ More
Access to previous results is of paramount importance in the scientific process. Recent progress in information management focuses on building e-infrastructures for the optimization of the research workflow, through both policy-driven and user-pulled dynamics. For decades, High-Energy Physics (HEP) has pioneered innovative solutions in the field of information management and dissemination. In light of a transforming information environment, it is important to assess the current usage of information resources by researchers and HEP provides a unique test-bed for this assessment. A survey of about 10% of practitioners in the field reveals usage trends and information needs. Community-based services, such as the pioneering arXiv and SPIRES systems, largely answer the need of the scientists, with a limited but increasing fraction of younger users relying on Google. Commercial services offered by publishers or database vendors are essentially unused in the field. The survey offers an insight into the most important features that users require to optimize their research workflow. These results inform the future evolution of information management in HEP and, as these researchers are traditionally ``early adopters'' of innovation in scholarly communication, can inspire developments of disciplinary repositories serving other communities.
△ Less
Submitted 22 April, 2008; v1 submitted 16 April, 2008;
originally announced April 2008.
-
Open Access Publishing in Particle Physics: A Brief Introduction for the non-Expert
Authors:
Travis C. Brooks
Abstract:
Open Access to particle physics literature does not sound particularly new or exciting, since particle physicists have been reading preprints for decades, and arXiv.org for 15 years. However new movements in Europe are attempting to make the peer-reviewed literature of the field fully Open Access. This is not a new movement, nor is it restricted to this field. However, given the field's history…
▽ More
Open Access to particle physics literature does not sound particularly new or exciting, since particle physicists have been reading preprints for decades, and arXiv.org for 15 years. However new movements in Europe are attempting to make the peer-reviewed literature of the field fully Open Access. This is not a new movement, nor is it restricted to this field. However, given the field's history of preprints and eprints, it is well suited to a change to a fully Open Access publishing model. Data shows that 90% of HEP published literature is freely available online, meaning that HEP libraries have little need for expensive journal subscriptions. As libraries begin to cancel journal subscriptions, the peer review process will lose its primary source of funding. Open Access publishing models can potentially address this issue. European physicists and funding agencies are proposing a consortium, SCOAP3, that might solve many of the objections to traditional Open Access publishing models in Particle Physics. These proposed changes should be viewed as a starting point for a serious look at the field's publication model, and are at least worthy of attention, if not adoption.
△ Less
Submitted 23 May, 2007;
originally announced May 2007.