Showing 1–19 of 19 results for author: Huynh, Q

LEGION: Harnessing Pre-trained Language Models for GitHub Topic Recommendations with Distribution-Balance Loss

Authors: Yen-Trang Dang, Thanh-Le Cong, Phuc-Thanh Nguyen, Anh M. T. Bui, Phuong T. Nguyen, Bach Le, Quyet-Thang Huynh

Abstract: Open-source development has revolutionized the software industry by promoting collaboration, transparency, and community-driven innovation. Today, a vast amount of various kinds of open-source software, which form networks of repositories, is often hosted on GitHub - a popular software development platform. To enhance the discoverability of the repository networks, i.e., groups of similar reposito… ▽ More Open-source development has revolutionized the software industry by promoting collaboration, transparency, and community-driven innovation. Today, a vast amount of various kinds of open-source software, which form networks of repositories, is often hosted on GitHub - a popular software development platform. To enhance the discoverability of the repository networks, i.e., groups of similar repositories, GitHub introduced repository topics in 2017 that enable users to more easily explore relevant projects by type, technology, and more. It is thus crucial to accurately assign topics for each GitHub repository. Current methods for automatic topic recommendation rely heavily on TF-IDF for encoding textual data, presenting challenges in understanding semantic nuances. This paper addresses the limitations of existing techniques by proposing Legion, a novel approach that leverages Pre-trained Language Models (PTMs) for recommending topics for GitHub repositories. The key novelty of Legion is three-fold. First, Legion leverages the extensive capabilities of PTMs in language understanding to capture contextual information and semantic meaning in GitHub repositories. Second, Legion overcomes the challenge of long-tailed distribution, which results in a bias toward popular topics in PTMs, by proposing a Distribution-Balanced Loss (DB Loss) to better train the PTMs. Third, Legion employs a filter to eliminate vague recommendations, thereby improving the precision of PTMs. Our empirical evaluation on a benchmark dataset of real-world GitHub repositories shows that Legion can improve vanilla PTMs by up to 26% on recommending GitHubs topics. Legion also can suggest GitHub topics more precisely and effectively than the state-of-the-art baseline with an average improvement of 20% and 5% in terms of Precision and F1-score, respectively. △ Less

Submitted 9 March, 2024; originally announced March 2024.

Comments: Accepted to EASE'24

AI for Anticipatory Action: Moving Beyond Climate Forecasting

Authors: Benjamin Q. Huynh, Mathew V. Kiang

Abstract: Disaster response agencies have been shifting from a paradigm of climate forecasting towards one of anticipatory action: assessing not just what the climate will be, but how it will impact specific populations, thereby enabling proactive response and resource allocation. Machine learning models are becoming exceptionally powerful at climate forecasting, but methodological gaps remain in terms of f… ▽ More Disaster response agencies have been shifting from a paradigm of climate forecasting towards one of anticipatory action: assessing not just what the climate will be, but how it will impact specific populations, thereby enabling proactive response and resource allocation. Machine learning models are becoming exceptionally powerful at climate forecasting, but methodological gaps remain in terms of facilitating anticipatory action. Here we provide an overview of anticipatory action, review relevant applications of machine learning, identify common challenges, and highlight areas where machine learning can uniquely contribute to advancing disaster response for populations most vulnerable to climate change. △ Less

Submitted 28 July, 2023; originally announced July 2023.

#REVAL: a semantic evaluation framework for hashtag recommendation

Authors: Areej Alsini, Du Q. Huynh, Amitava Datta

Abstract: Automatic evaluation of hashtag recommendation models is a fundamental task in many online social network systems. In the traditional evaluation method, the recommended hashtags from an algorithm are firstly compared with the ground truth hashtags for exact correspondences. The number of exact matches is then used to calculate the hit rate, hit ratio, precision, recall, or F1-score. This way of ev… ▽ More Automatic evaluation of hashtag recommendation models is a fundamental task in many online social network systems. In the traditional evaluation method, the recommended hashtags from an algorithm are firstly compared with the ground truth hashtags for exact correspondences. The number of exact matches is then used to calculate the hit rate, hit ratio, precision, recall, or F1-score. This way of evaluating hashtag similarities is inadequate as it ignores the semantic correlation between the recommended and ground truth hashtags. To tackle this problem, we propose a novel semantic evaluation framework for hashtag recommendation, called #REval. This framework includes an internal module referred to as BERTag, which automatically learns the hashtag embeddings. We investigate on how the #REval framework performs under different word embedding methods and different numbers of synonyms and hashtags in the recommendation using our proposed #REval-hit-ratio measure. Our experiments of the proposed framework on three large datasets show that #REval gave more meaningful hashtag synonyms for hashtag recommendation evaluation. Our analysis also highlights the sensitivity of the framework to the word embedding technique, with #REval based on BERTag more superior over #REval based on FastText and Word2Vec. △ Less

Submitted 24 May, 2023; originally announced May 2023.

Comments: 18 pages, 4 figures

ACM Class: I.2.7

Potential for allocative harm in an environmental justice data tool

Authors: Benjamin Q. Huynh, Elizabeth T. Chin, Allison Koenecke, Derek Ouyang, Daniel E. Ho, Mathew V. Kiang, David H. Rehkopf

Abstract: Neighborhood-level screening algorithms are increasingly being deployed to inform policy decisions. We evaluate one such algorithm, CalEnviroScreen - designed to promote environmental justice and used to guide hundreds of millions of dollars in public funding annually - assessing its potential for allocative harm. We observe the model to be sensitive to subjective model decisions, with 16% of trac… ▽ More Neighborhood-level screening algorithms are increasingly being deployed to inform policy decisions. We evaluate one such algorithm, CalEnviroScreen - designed to promote environmental justice and used to guide hundreds of millions of dollars in public funding annually - assessing its potential for allocative harm. We observe the model to be sensitive to subjective model decisions, with 16% of tracts potentially changing designation, as well as financially consequential, estimating the effect of its positive designations as a 104% (62-145%) increase in funding, equivalent to \$2.08 billion (\$1.56-2.41 billion) over four years. We also observe allocative tradeoffs and susceptibility to manipulation, raising ethical concerns. We recommend incorporating sensitivity analyses to mitigate allocative harm and accountability mechanisms to prevent misuse. △ Less

Submitted 12 April, 2023; v1 submitted 12 April, 2023; originally announced April 2023.

Journal ref: Nat Mach Intell 6, 187-194 (2024)

Efficient learning of large sets of locally optimal classification rules

Authors: Van Quoc Phuong Huynh, Johannes Fürnkranz, Florian Beck

Abstract: Conventional rule learning algorithms aim at finding a set of simple rules, where each rule covers as many examples as possible. In this paper, we argue that the rules found in this way may not be the optimal explanations for each of the examples they cover. Instead, we propose an efficient algorithm that aims at finding the best rule covering each training example in a greedy optimization consist… ▽ More Conventional rule learning algorithms aim at finding a set of simple rules, where each rule covers as many examples as possible. In this paper, we argue that the rules found in this way may not be the optimal explanations for each of the examples they cover. Instead, we propose an efficient algorithm that aims at finding the best rule covering each training example in a greedy optimization consisting of one specialization and one generalization loop. These locally optimal rules are collected and then filtered for a final rule set, which is much larger than the sets learned by conventional rule learning algorithms. A new example is classified by selecting the best among the rules that cover this example. In our experiments on small to very large datasets, the approach's average classification accuracy is higher than that of state-of-the-art rule learning algorithms. Moreover, the algorithm is highly efficient and can inherently be processed in parallel without affecting the learned rule set and so the classification accuracy. We thus believe that it closes an important gap for large-scale classification rule induction. △ Less

Submitted 26 January, 2023; v1 submitted 24 January, 2023; originally announced January 2023.

Comments: article, 40 pages, Machine Learning journal (2023)

Journal ref: Mach Learn (2023)

Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning

Authors: Thanh Le-Cong, Duc-Minh Luong, Xuan Bach D. Le, David Lo, Nhat-Hoa Tran, Bui Quang-Huy, Quyet-Thang Huynh

Abstract: Automated program repair (APR) faces the challenge of test overfitting, where generated patches pass validation tests but fail to generalize. Existing methods for patch assessment involve generating new tests or manual inspection, which can be time-consuming or biased. In this paper, we propose a novel technique, INVALIDATOR, to automatically assess the correctness of APR-generated patches via sem… ▽ More Automated program repair (APR) faces the challenge of test overfitting, where generated patches pass validation tests but fail to generalize. Existing methods for patch assessment involve generating new tests or manual inspection, which can be time-consuming or biased. In this paper, we propose a novel technique, INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR leverages program invariants to reason about program semantics while also capturing program syntax through language semantics learned from a large code corpus using a pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that an APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains erroneous behaviors from the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is threefold. First, INVALIDATOR leverages both semantic and syntactic reasoning to enhance its discriminative capability. Second, INVALIDATOR does not require new test cases to be generated, but instead only relies on the current test suite and uses invariant inference to generalize program behaviors. Third, INVALIDATOR is fully automated. Experimental results demonstrate that INVALIDATOR outperforms existing methods in terms of Accuracy and F-measure, correctly identifying 79% of overfitting patches and detecting 23% more overfitting patches than the best baseline. △ Less

Submitted 17 March, 2023; v1 submitted 3 January, 2023; originally announced January 2023.

Journal ref: IEEE Transactions on Software Engineering, 2023

Usability and Aesthetics: Better Together for Automated Repair of Web Pages

Authors: Thanh Le-Cong, Xuan Bach D. Le, Quyet-Thang Huynh, Phi-Le Nguyen

Abstract: With the recent explosive growth of mobile devices such as smartphones or tablets, guaranteeing consistent web appearance across all environments has become a significant problem. This happens simply because it is hard to keep track of the web appearance on different sizes and types of devices that render the web pages. Therefore, fixing the inconsistent appearance of web pages can be difficult, a… ▽ More With the recent explosive growth of mobile devices such as smartphones or tablets, guaranteeing consistent web appearance across all environments has become a significant problem. This happens simply because it is hard to keep track of the web appearance on different sizes and types of devices that render the web pages. Therefore, fixing the inconsistent appearance of web pages can be difficult, and the cost incurred can be huge, e.g., poor user experience and financial loss due to it. Recently, automated web repair techniques have been proposed to automatically resolve inconsistent web page appearance, focusing on improving usability. However, generated patches tend to disrupt the webpage's layout, rendering the repaired webpage aesthetically unpleasing, e.g., distorted images or misalignment of components. In this paper, we propose an automated repair approach for web pages based on meta-heuristic algorithms that can assure both usability and aesthetics. The key novelty that empowers our approach is a novel fitness function that allows us to optimistically evolve buggy web pages to find the best solution that optimizes both usability and aesthetics at the same time. Empirical evaluations show that our approach is able to successfully resolve mobile-friendly problems in 94% of the evaluation subjects, significantly outperforming state-of-the-art baseline techniques in terms of both usability and aesthetics. △ Less

Submitted 1 January, 2022; originally announced January 2022.

Comments: Accepted to ISSRE 2021, Research Track

Journal ref: 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE 2021)

Toward the Analysis of Graph Neural Networks

Authors: Thanh-Dat Nguyen, Thanh Le-Cong, ThanhVu H. Nguyen, Xuan-Bach D. Le, Quyet-Thang Huynh

Abstract: Graph Neural Networks (GNNs) have recently emerged as a robust framework for graph-structured data. They have been applied to many problems such as knowledge graph analysis, social networks recommendation, and even Covid19 detection and vaccine developments. However, unlike other deep neural networks such as Feed Forward Neural Networks (FFNNs), few analyses such as verification and property infer… ▽ More Graph Neural Networks (GNNs) have recently emerged as a robust framework for graph-structured data. They have been applied to many problems such as knowledge graph analysis, social networks recommendation, and even Covid19 detection and vaccine developments. However, unlike other deep neural networks such as Feed Forward Neural Networks (FFNNs), few analyses such as verification and property inferences exist, potentially due to dynamic behaviors of GNNs, which can take arbitrary graphs as input, whereas FFNNs which only take fixed size numerical vectors as inputs. This paper proposes an approach to analyze GNNs by converting them into FFNNs and reusing existing FFNNs analyses. We discuss various designs to ensure the scalability and accuracy of the conversions. We illustrate our method on a study case of node classification. We believe that our approach opens new research directions for understanding and analyzing GNNs. △ Less

Submitted 31 December, 2021; originally announced January 2022.

Comments: Accepted to ICSE 2022, NIER track

Journal ref: The 44th IEEE/ACM International Conference on Software Engineering (ICSE 2022)

Incomplete Knowledge Graph Alignment

Authors: Vinh Van Tong, Thanh Trung Huynh, Thanh Tam Nguyen, Hongzhi Yin, Quoc Viet Hung Nguyen, Quyet Thang Huynh

Abstract: Knowledge graph (KG) alignment - the task of recognizing entities referring to the same thing in different KGs - is recognized as one of the most important operations in the field of KG construction and completion. However, existing alignment techniques often assume that the input KGs are complete and isomorphic, which is not true due to the real-world heterogeneity in the domain, size, and sparsi… ▽ More Knowledge graph (KG) alignment - the task of recognizing entities referring to the same thing in different KGs - is recognized as one of the most important operations in the field of KG construction and completion. However, existing alignment techniques often assume that the input KGs are complete and isomorphic, which is not true due to the real-world heterogeneity in the domain, size, and sparsity. In this work, we address the problem of aligning incomplete KGs with representation learning. Our KG embedding framework exploits two feature channels: transitivity-based and proximity-based. The former captures the consistency constraints between entities via translation paths, while the latter captures the neighbourhood structure of KGs via attention guided relation-aware graph neural network. The two feature channels are jointly learned to exchange important features between the input KGs while enforcing the output representations of the input KGs in the same embedding space. Also, we develop a missing links detector that discovers and recovers the missing links in the input KGs during the training process, which helps mitigate the incompleteness issue and thus improve the compatibility of the learned representations. The embeddings then are fused to generate the alignment result, and the high-confidence matched node pairs are updated to the pre-aligned supervision data to improve the embeddings gradually. Empirical results show that our model is more accurate than the SOTA and is robust against different levels of incompleteness. △ Less

Submitted 15 March, 2022; v1 submitted 16 December, 2021; originally announced December 2021.

Communication-hiding pipelined BiCGSafe methods for solving large linear systems

Authors: Viet Q. H. Huynh, Hiroshi Suito

Abstract: Recently, a new variant of the BiCGStab method, known as the pipeline BiCGStab, has been proposed. This method can achieve a higher degree of scalability and speed-up rates through a mechanism in which the communication phase for the computation of the inner product can be overlapped with the computation of the matrix-vector product. On the other hand, there exist several generalized iteration met… ▽ More Recently, a new variant of the BiCGStab method, known as the pipeline BiCGStab, has been proposed. This method can achieve a higher degree of scalability and speed-up rates through a mechanism in which the communication phase for the computation of the inner product can be overlapped with the computation of the matrix-vector product. On the other hand, there exist several generalized iteration methods with better convergence behavior than BiCGStab such as ssBiCGSafe, BiCGSafe, GPBi-CG. Of these methods, ssBiCGSafe, which requires a single phase of computing inner products per one iteration, is best suited for high-performance computing systems. In this paper, inspired by the success of the pipelined BiCGStab method, we propose variations of the ssBiCGSafe method, in which only one phase of inner product computation per iteration is required and this inner product computation phase can be overlapped with the matrix-vector computation. Through numerical experiments, we show that the proposed methods lead to improvements in convergence behavior and execution time compared to the pipelined BiCGStab and ssBiCGSafe methods. △ Less

Submitted 27 October, 2021; v1 submitted 24 August, 2021; originally announced August 2021.

VinDr-SpineXR: A deep learning framework for spinal lesions detection and classification from radiographs

Authors: Hieu T. Nguyen, Hieu H. Pham, Nghia T. Nguyen, Ha Q. Nguyen, Thang Q. Huynh, Minh Dao, Van Vu

Abstract: Radiographs are used as the most important imaging tool for identifying spine anomalies in clinical practice. The evaluation of spinal bone lesions, however, is a challenging task for radiologists. This work aims at developing and evaluating a deep learning-based framework, named VinDr-SpineXR, for the classification and localization of abnormalities from spine X-rays. First, we build a large data… ▽ More Radiographs are used as the most important imaging tool for identifying spine anomalies in clinical practice. The evaluation of spinal bone lesions, however, is a challenging task for radiologists. This work aims at developing and evaluating a deep learning-based framework, named VinDr-SpineXR, for the classification and localization of abnormalities from spine X-rays. First, we build a large dataset, comprising 10,468 spine X-ray images from 5,000 studies, each of which is manually annotated by an experienced radiologist with bounding boxes around abnormal findings in 13 categories. Using this dataset, we then train a deep learning classifier to determine whether a spine scan is abnormal and a detector to localize 7 crucial findings amongst the total 13. The VinDr-SpineXR is evaluated on a test set of 2,078 images from 1,000 studies, which is kept separate from the training set. It demonstrates an area under the receiver operating characteristic curve (AUROC) of 88.61% (95% CI 87.19%, 90.02%) for the image-level classification task and a mean average precision ([email protected]) of 33.56% for the lesion-level localization task. These results serve as a proof of concept and set a baseline for future research in this direction. To encourage advances, the dataset, codes, and trained deep learning models are made publicly available. △ Less

Submitted 24 June, 2021; originally announced June 2021.

Comments: This is a preprint of our paper which was accepted for publication by the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021)

Scene Gated Social Graph: Pedestrian Trajectory Prediction Based on Dynamic Social Graphs and Scene Constraints

Authors: Hao Xue, Du Q. Huynh, Mark Reynolds

Abstract: Pedestrian trajectory prediction is valuable for understanding human motion behaviors and it is challenging because of the social influence from other pedestrians, the scene constraints and the multimodal possibilities of predicted trajectories. Most existing methods only focus on two of the above three key elements. In order to jointly consider all these elements, we propose a novel trajectory pr… ▽ More Pedestrian trajectory prediction is valuable for understanding human motion behaviors and it is challenging because of the social influence from other pedestrians, the scene constraints and the multimodal possibilities of predicted trajectories. Most existing methods only focus on two of the above three key elements. In order to jointly consider all these elements, we propose a novel trajectory prediction method named Scene Gated Social Graph (SGSG). In the proposed SGSG, dynamic graphs are used to describe the social relationship among pedestrians. The social and scene influences are taken into account through the scene gated social graph features which combine the encoded social graph features and semantic scene features. In addition, a VAE module is incorporated to learn the scene gated social feature and sample latent variables for generating multiple trajectories that are socially and environmentally acceptable. We compare our SGSG against twenty state-of-the-art pedestrian trajectory prediction methods and the results show that the proposed method achieves superior performance on two widely used trajectory prediction benchmarks. △ Less

Submitted 12 October, 2020; originally announced October 2020.

Artificial intelligence supported anemia control system (AISACS) to prevent anemia in maintenance hemodialysis patients

Authors: Toshiaki Ohara, Hiroshi Ikeda, Yoshiki Sugitani, Hiroshi Suito, Viet Quang Huy Huynh, Masaru Kinomura, Soichiro Haraguchi, Kazufumi Sakurama

Abstract: Anemia, for which erythropoiesis-stimulating agents (ESAs) and iron supplements (ISs) are used as preventive measures, presents important difficulties for hemodialysis patients. Nevertheless, the number of physicians able to manage such medications appropriately is not keeping pace with the rapid increase of hemodialysis patients. Moreover, the high cost of ESAs imposes heavy burdens on medical in… ▽ More Anemia, for which erythropoiesis-stimulating agents (ESAs) and iron supplements (ISs) are used as preventive measures, presents important difficulties for hemodialysis patients. Nevertheless, the number of physicians able to manage such medications appropriately is not keeping pace with the rapid increase of hemodialysis patients. Moreover, the high cost of ESAs imposes heavy burdens on medical insurance systems. An artificial-intelligence-supported anemia control system (AISACS) trained using administration direction data from experienced physicians has been developed by the authors. For the system, appropriate data selection and rectification techniques play important roles. Decision making related to ESAs poses a multi-class classification problem for which a two-step classification technique is introduced. Several validations have demonstrated that AISACS exhibits high performance with correct classification rates of 72-87% and clinically appropriate classification rates of 92-98%. △ Less

Submitted 5 October, 2020; originally announced October 2020.

Comments: 14 pages and 7 figures

ACM Class: H.4.2; I.2.1; I.5.4; J.3

Hit ratio: An Evaluation Metric for Hashtag Recommendation

Authors: Areej Alsini, Du Q. Huynh, Amitava Datta

Abstract: Hashtag recommendation is a crucial task, especially with an increase of interest in using social media platforms such as Twitter in the last decade. Hashtag recommendation systems automatically suggest hashtags to a user while writing a tweet. Most of the research in the area of hashtag recommendation have used classical metrics such as hit rate, precision, recall, and F1-score to measure the acc… ▽ More Hashtag recommendation is a crucial task, especially with an increase of interest in using social media platforms such as Twitter in the last decade. Hashtag recommendation systems automatically suggest hashtags to a user while writing a tweet. Most of the research in the area of hashtag recommendation have used classical metrics such as hit rate, precision, recall, and F1-score to measure the accuracy of hashtag recommendation systems. These metrics are based on the exact match of the recommended hashtags with their corresponding ground truth. However, it is not clear how adequate these metrics to evaluate hashtag recommendation. The research question that we are interested in seeking an answer is: are these metrics adequate for evaluating hashtag recommendation systems when the numbers of ground truth hashtags in tweets are highly variable? In this paper, we propose a new metric which we call hit ratio for hashtag recommendation. Extensive evaluation through hypothetical examples and real-world application across a range of hashtag recommendation models indicate that the hit ratio is a useful metric. A comparison of hit ratio with the classical evaluation metrics reveals their limitations. △ Less

Submitted 2 October, 2020; originally announced October 2020.

Take a NAP: Non-Autoregressive Prediction for Pedestrian Trajectories

Authors: Hao Xue, Du. Q. Huynh, Mark Reynolds

Abstract: Pedestrian trajectory prediction is a challenging task as there are three properties of human movement behaviors which need to be addressed, namely, the social influence from other pedestrians, the scene constraints, and the multimodal (multiroute) nature of predictions. Although existing methods have explored these key properties, the prediction process of these methods is autoregressive. This me… ▽ More Pedestrian trajectory prediction is a challenging task as there are three properties of human movement behaviors which need to be addressed, namely, the social influence from other pedestrians, the scene constraints, and the multimodal (multiroute) nature of predictions. Although existing methods have explored these key properties, the prediction process of these methods is autoregressive. This means they can only predict future locations sequentially. In this paper, we present NAP, a non-autoregressive method for trajectory prediction. Our method comprises specifically designed feature encoders and a latent variable generator to handle the three properties above. It also has a time-agnostic context generator and a time-specific context generator for non-autoregressive prediction. Through extensive experiments that compare NAP against several recent methods, we show that NAP has state-of-the-art trajectory prediction performance. △ Less

Submitted 21 April, 2020; originally announced April 2020.

Loss Switching Fusion with Similarity Search for Video Classification

Authors: Lei Wang, Du Q. Huynh, Moussa Reda Mansour

Abstract: From video streaming to security and surveillance applications, video data play an important role in our daily living today. However, managing a large amount of video data and retrieving the most useful information for the user remain a challenging task. In this paper, we propose a novel video classification system that would benefit the scene understanding task. We define our classification probl… ▽ More From video streaming to security and surveillance applications, video data play an important role in our daily living today. However, managing a large amount of video data and retrieving the most useful information for the user remain a challenging task. In this paper, we propose a novel video classification system that would benefit the scene understanding task. We define our classification problem as classifying background and foreground motions using the same feature representation for outdoor scenes. This means that the feature representation needs to be robust enough and adaptable to different classification tasks. We propose a lightweight Loss Switching Fusion Network (LSFNet) for the fusion of spatiotemporal descriptors and a similarity search scheme with soft voting to boost the classification performance. The proposed system has a variety of potential applications such as content-based video clustering, video filtering, etc. Evaluation results on two private industry datasets show that our system is robust in both classifying different background motions and detecting human motions from these background motions. △ Less

Submitted 27 June, 2019; originally announced June 2019.

Comments: Accepted by ICIP 2019

A Comparative Review of Recent Kinect-based Action Recognition Algorithms

Authors: Lei Wang, Du Q. Huynh, Piotr Koniusz

Abstract: Video-based human action recognition is currently one of the most active research areas in computer vision. Various research studies indicate that the performance of action recognition is highly dependent on the type of features being extracted and how the actions are represented. Since the release of the Kinect camera, a large number of Kinect-based human action recognition techniques have been p… ▽ More Video-based human action recognition is currently one of the most active research areas in computer vision. Various research studies indicate that the performance of action recognition is highly dependent on the type of features being extracted and how the actions are represented. Since the release of the Kinect camera, a large number of Kinect-based human action recognition techniques have been proposed in the literature. However, there still does not exist a thorough comparison of these Kinect-based techniques under the grouping of feature types, such as handcrafted versus deep learning features and depth-based versus skeleton-based features. In this paper, we analyze and compare ten recent Kinect-based algorithms for both cross-subject action recognition and cross-view action recognition using six benchmark datasets. In addition, we have implemented and improved some of these techniques and included their variants in the comparison. Our experiments show that the majority of methods perform better on cross-subject action recognition than cross-view action recognition, that skeleton-based features are more robust for cross-view recognition than depth-based features, and that deep learning features are suitable for large datasets. △ Less

Submitted 24 June, 2019; originally announced June 2019.

Comments: Accepted by the IEEE Transactions on Image Processing

Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition with CNNs

Authors: Lei Wang, Piotr Koniusz, Du Q. Huynh

Abstract: In this paper, we revive the use of old-fashioned handcrafted video representations for action recognition and put new life into these techniques via a CNN-based hallucination step. Despite of the use of RGB and optical flow frames, the I3D model (amongst others) thrives on combining its output with the Improved Dense Trajectory (IDT) and extracted with its low-level video descriptors encoded via… ▽ More In this paper, we revive the use of old-fashioned handcrafted video representations for action recognition and put new life into these techniques via a CNN-based hallucination step. Despite of the use of RGB and optical flow frames, the I3D model (amongst others) thrives on combining its output with the Improved Dense Trajectory (IDT) and extracted with its low-level video descriptors encoded via Bag-of-Words (BoW) and Fisher Vectors (FV). Such a fusion of CNNs and handcrafted representations is time-consuming due to pre-processing, descriptor extraction, encoding and tuning parameters. Thus, we propose an end-to-end trainable network with streams which learn the IDT-based BoW/FV representations at the training stage and are simple to integrate with the I3D model. Specifically, each stream takes I3D feature maps ahead of the last 1D conv. layer and learns to `translate' these maps to BoW/FV representations. Thus, our model can hallucinate and use such synthesized BoW/FV representations at the testing stage. We show that even features of the entire I3D optical flow stream can be hallucinated thus simplifying the pipeline. Our model saves 20-55h of computations and yields state-of-the-art results on four publicly available datasets. △ Less

Submitted 18 August, 2019; v1 submitted 13 June, 2019; originally announced June 2019.

Comments: First two authors contributed equally. This paper is accepted by ICCV'19

Journal ref: ICCV 2019

HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition

Authors: Hossein Rahmani, Arif Mahmood, Du Q. Huynh, Ajmal Mian

Abstract: Existing techniques for 3D action recognition are sensitive to viewpoint variations because they extract features from depth images which change significantly with viewpoint. In contrast, we directly process the pointclouds and propose a new technique for action recognition which is more robust to noise, action speed and viewpoint variations. Our technique consists of a novel descriptor and keypoi… ▽ More Existing techniques for 3D action recognition are sensitive to viewpoint variations because they extract features from depth images which change significantly with viewpoint. In contrast, we directly process the pointclouds and propose a new technique for action recognition which is more robust to noise, action speed and viewpoint variations. Our technique consists of a novel descriptor and keypoint detection algorithm. The proposed descriptor is extracted at a point by encoding the Histogram of Oriented Principal Components (HOPC) within an adaptive spatio-temporal support volume around that point. Based on this descriptor, we present a novel method to detect Spatio-Temporal Key-Points (STKPs) in 3D pointcloud sequences. Experimental results show that the proposed descriptor and STKP detector outperform state-of-the-art algorithms on three benchmark human activity datasets. We also introduce a new multiview public dataset and show the robustness of our proposed method to viewpoint variations. △ Less

Submitted 22 September, 2014; v1 submitted 17 August, 2014; originally announced August 2014.

Comments: ECCV 2014