-
Comparison of Large Language Models for Generating Contextually Relevant Questions
Authors:
Ivo Lodovico Molina,
Valdemar Švábenský,
Tsubasa Minematsu,
Li Chen,
Fumiya Okubo,
Atsushi Shimada
Abstract:
This study explores the effectiveness of Large Language Models (LLMs) for Automatic Question Generation in educational settings. Three LLMs are compared in their ability to create questions from university slide text without fine-tuning. Questions were obtained in a two-step pipeline: first, answer phrases were extracted from slides using Llama 2-Chat 13B; then, the three models generated question…
▽ More
This study explores the effectiveness of Large Language Models (LLMs) for Automatic Question Generation in educational settings. Three LLMs are compared in their ability to create questions from university slide text without fine-tuning. Questions were obtained in a two-step pipeline: first, answer phrases were extracted from slides using Llama 2-Chat 13B; then, the three models generated questions for each answer. To analyze whether the questions would be suitable in educational applications for students, a survey was conducted with 46 students who evaluated a total of 246 questions across five metrics: clarity, relevance, difficulty, slide relation, and question-answer alignment. Results indicate that GPT-3.5 and Llama 2-Chat 13B outperform Flan T5 XXL by a small margin, particularly in terms of clarity and question-answer alignment. GPT-3.5 especially excels at tailoring questions to match the input answers. The contribution of this research is the analysis of the capacity of LLMs for Automatic Question Generation in education.
△ Less
Submitted 30 July, 2024;
originally announced July 2024.
-
E2Vec: Feature Embedding with Temporal Information for Analyzing Student Actions in E-Book Systems
Authors:
Yuma Miyazaki,
Valdemar Švábenský,
Yuta Taniguchi,
Fumiya Okubo,
Tsubasa Minematsu,
Atsushi Shimada
Abstract:
Digital textbook (e-book) systems record student interactions with textbooks as a sequence of events called EventStream data. In the past, researchers extracted meaningful features from EventStream, and utilized them as inputs for downstream tasks such as grade prediction and modeling of student behavior. Previous research evaluated models that mainly used statistical-based features derived from E…
▽ More
Digital textbook (e-book) systems record student interactions with textbooks as a sequence of events called EventStream data. In the past, researchers extracted meaningful features from EventStream, and utilized them as inputs for downstream tasks such as grade prediction and modeling of student behavior. Previous research evaluated models that mainly used statistical-based features derived from EventStream logs, such as the number of operation types or access frequencies. While these features are useful for providing certain insights, they lack temporal information that captures fine-grained differences in learning behaviors among different students. This study proposes E2Vec, a novel feature representation method based on word embeddings. The proposed method regards operation logs and their time intervals for each student as a string sequence of characters and generates a student vector of learning activity features that incorporates time information. We applied fastText to generate an embedding vector for each of 305 students in a dataset from two years of computer science courses. Then, we investigated the effectiveness of E2Vec in an at-risk detection task, demonstrating potential for generalizability and performance.
△ Less
Submitted 24 May, 2024;
originally announced July 2024.
-
Evaluating Algorithmic Bias in Models for Predicting Academic Performance of Filipino Students
Authors:
Valdemar Švábenský,
Mélina Verger,
Maria Mercedes T. Rodrigo,
Clarence James G. Monterozo,
Ryan S. Baker,
Miguel Zenon Nicanor Lerias Saavedra,
Sébastien Lallé,
Atsushi Shimada
Abstract:
Algorithmic bias is a major issue in machine learning models in educational contexts. However, it has not yet been studied thoroughly in Asian learning contexts, and only limited work has considered algorithmic bias based on regional (sub-national) background. As a step towards addressing this gap, this paper examines the population of 5,986 students at a large university in the Philippines, inves…
▽ More
Algorithmic bias is a major issue in machine learning models in educational contexts. However, it has not yet been studied thoroughly in Asian learning contexts, and only limited work has considered algorithmic bias based on regional (sub-national) background. As a step towards addressing this gap, this paper examines the population of 5,986 students at a large university in the Philippines, investigating algorithmic bias based on students' regional background. The university used the Canvas learning management system (LMS) in its online courses across a broad range of domains. Over the period of three semesters, we collected 48.7 million log records of the students' activity in Canvas. We used these logs to train binary classification models that predict student grades from the LMS activity. The best-performing model reached AUC of 0.75 and weighted F1-score of 0.79. Subsequently, we examined the data for bias based on students' region. Evaluation using three metrics: AUC, weighted F1-score, and MADD showed consistent results across all demographic groups. Thus, no unfairness was observed against a particular student group in the grade predictions.
△ Less
Submitted 15 July, 2024; v1 submitted 16 May, 2024;
originally announced May 2024.
-
Gaze-Based Intention Recognition for Human-Robot Collaboration
Authors:
Valerio Belcamino,
Miwa Takase,
Mariya Kilina,
Alessandro Carfì,
Akira Shimada,
Sota Shimizu,
Fulvio Mastrogiovanni
Abstract:
This work aims to tackle the intent recognition problem in Human-Robot Collaborative assembly scenarios. Precisely, we consider an interactive assembly of a wooden stool where the robot fetches the pieces in the correct order and the human builds the parts following the instruction manual. The intent recognition is limited to the idle state estimation and it is needed to ensure a better synchroniz…
▽ More
This work aims to tackle the intent recognition problem in Human-Robot Collaborative assembly scenarios. Precisely, we consider an interactive assembly of a wooden stool where the robot fetches the pieces in the correct order and the human builds the parts following the instruction manual. The intent recognition is limited to the idle state estimation and it is needed to ensure a better synchronization between the two agents. We carried out a comparison between two distinct solutions involving wearable sensors and eye tracking integrated into the perception pipeline of a flexible planning architecture based on Hierarchical Task Networks. At runtime, the wearable sensing module exploits the raw measurements from four 9-axis Inertial Measurement Units positioned on the wrists and hands of the user as an input for a Long Short-Term Memory Network. On the other hand, the eye tracking relies on a Head Mounted Display and Unreal Engine.
We tested the effectiveness of the two approaches with 10 participants, each of whom explored both options in alternate order. We collected explicit metrics about the attractiveness and efficiency of the two techniques through User Experience Questionnaires as well as implicit criteria regarding the classification time and the overall assembly time.
The results of our work show that the two methods can reach comparable performances both in terms of effectiveness and user preference. Future development could aim at joining the two approaches two allow the recognition of more complex activities and to anticipate the user actions.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Hand Orientation Estimation in Probability Density Form
Authors:
Kazuaki Kondo,
Daisuke Deguchi,
Atsushi Shimada
Abstract:
Hand orientation is an essential feature required to understand hand behaviors and subsequently support human activities. In this paper, we present a new method for estimating hand orientation in probability density form. It can solve the cyclicity problem in direct angular representation and enables the integration of multiple predictions based on different features. We validated the performance…
▽ More
Hand orientation is an essential feature required to understand hand behaviors and subsequently support human activities. In this paper, we present a new method for estimating hand orientation in probability density form. It can solve the cyclicity problem in direct angular representation and enables the integration of multiple predictions based on different features. We validated the performance of the proposed method and an integration example using our dataset, which captured cooperative group work.
△ Less
Submitted 12 June, 2019;
originally announced June 2019.
-
TransCut: Transparent Object Segmentation from a Light-Field Image
Authors:
Yichao Xu,
Hajime Nagahara,
Atsushi Shimada,
Rin-ichiro Taniguchi
Abstract:
The segmentation of transparent objects can be very useful in computer vision applications. However, because they borrow texture from their background and have a similar appearance to their surroundings, transparent objects are not handled well by regular image segmentation methods. We propose a method that overcomes these problems using the consistency and distortion properties of a light-field i…
▽ More
The segmentation of transparent objects can be very useful in computer vision applications. However, because they borrow texture from their background and have a similar appearance to their surroundings, transparent objects are not handled well by regular image segmentation methods. We propose a method that overcomes these problems using the consistency and distortion properties of a light-field image. Graph-cut optimization is applied for the pixel labeling problem. The light-field linearity is used to estimate the likelihood of a pixel belonging to the transparent object or Lambertian background, and the occlusion detector is used to find the occlusion boundary. We acquire a light field dataset for the transparent object, and use this dataset to evaluate our method. The results demonstrate that the proposed method successfully segments transparent objects from the background.
△ Less
Submitted 21 November, 2015;
originally announced November 2015.
