Showing 1–17 of 17 results for author: Ban, X

Data Poisoning Attacks in Intelligent Transportation Systems: A Survey

Authors: Feilong Wang, Xin Wang, Xuegang Ban

Abstract: Emerging technologies drive the ongoing transformation of Intelligent Transportation Systems (ITS). This transformation has given rise to cybersecurity concerns, among which data poisoning attack emerges as a new threat as ITS increasingly relies on data. In data poisoning attacks, attackers inject malicious perturbations into datasets, potentially leading to inaccurate results in offline learning… ▽ More Emerging technologies drive the ongoing transformation of Intelligent Transportation Systems (ITS). This transformation has given rise to cybersecurity concerns, among which data poisoning attack emerges as a new threat as ITS increasingly relies on data. In data poisoning attacks, attackers inject malicious perturbations into datasets, potentially leading to inaccurate results in offline learning and real-time decision-making processes. This paper concentrates on data poisoning attack models against ITS. We identify the main ITS data sources vulnerable to poisoning attacks and application scenarios that enable staging such attacks. A general framework is developed following rigorous study process from cybersecurity but also considering specific ITS application needs. Data poisoning attacks against ITS are reviewed and categorized following the framework. We then discuss the current limitations of these attack models and the future research directions. Our work can serve as a guideline to better understand the threat of data poisoning attacks against ITS applications, while also giving a perspective on the future development of trustworthy ITS. △ Less

Submitted 5 July, 2024; originally announced July 2024.

Comments: This is an accepted version that is published on journal Transportation Research Part C: Emerging Technologies

Mitigating biases in big mobility data: a case study of monitoring large-scale transit systems

Authors: Feilong Wang, Xuegang Ban, Peng Chen, Chenxi Liu, Rong Zhao

Abstract: Big mobility datasets (BMD) have shown many advantages in studying human mobility and evaluating the performance of transportation systems. However, the quality of BMD remains poorly understood. This study evaluates biases in BMD and develops mitigation methods. Using Google and Apple mobility data as examples, this study compares them with benchmark data from governmental agencies. Spatio-tempora… ▽ More Big mobility datasets (BMD) have shown many advantages in studying human mobility and evaluating the performance of transportation systems. However, the quality of BMD remains poorly understood. This study evaluates biases in BMD and develops mitigation methods. Using Google and Apple mobility data as examples, this study compares them with benchmark data from governmental agencies. Spatio-temporal discrepancies between BMD and benchmark are observed and their impacts on transportation applications are investigated, emphasizing the urgent need to address these biases to prevent misguided policymaking. This study further proposes and tests a bias mitigation method. It is shown that the mitigated BMD could generate valuable insights into large-scale public transit systems across 100+ US counties, revealing regional disparities of the recovery of transit systems from the COVID-19. This study underscores the importance of caution when using BMD in transportation research and presents effective mitigation strategies that would benefit practitioners. △ Less

Submitted 15 July, 2024; originally announced July 2024.

Comments: 25 pages, 10 figures. Transportation Letters. August 2024

A Two-stage Evolutionary Framework For Multi-objective Optimization

Authors: Peng Chen, Jing Liang, Kangjia Qiao, Ponnuthurai Nagaratnam Suganthan, Xuanxuan Ban

Abstract: In the field of evolutionary multi-objective optimization, the approximation of the Pareto front (PF) is achieved by utilizing a collection of representative candidate solutions that exhibit desirable convergence and diversity. Although several multi-objective evolutionary algorithms (MOEAs) have been designed, they still have difficulties in keeping balance between convergence and diversity of po… ▽ More In the field of evolutionary multi-objective optimization, the approximation of the Pareto front (PF) is achieved by utilizing a collection of representative candidate solutions that exhibit desirable convergence and diversity. Although several multi-objective evolutionary algorithms (MOEAs) have been designed, they still have difficulties in keeping balance between convergence and diversity of population. To better solve multi-objective optimization problems (MOPs), this paper proposes a Two-stage Evolutionary Framework For Multi-objective Optimization (TEMOF). Literally, algorithms are divided into two stages to enhance the search capability of the population. During the initial half of evolutions, parental selection is exclusively conducted from the primary population. Additionally, we not only perform environmental selection on the current population, but we also establish an external archive to store individuals situated on the first PF. Subsequently, in the second stage, parents are randomly chosen either from the population or the archive. In the experiments, one classic MOEA and two state-of-the-art MOEAs are integrated into the framework to form three new algorithms. The experimental results demonstrate the superior and robust performance of the proposed framework across a wide range of MOPs. Besides, the winner among three new algorithms is compared with several existing MOEAs and shows better results. Meanwhile, we conclude the reasons that why the two-stage framework is effect for the existing benchmark functions. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: Accepted by the CEC conference of WCCI2024

Modeling User Fatigue for Sequential Recommendation

Authors: Nian Li, Xin Ban, Cheng Ling, Chen Gao, Lantao Hu, Peng Jiang, Kun Gai, Yong Li, Qingmin Liao

Abstract: Recommender systems filter out information that meets user interests. However, users may be tired of the recommendations that are too similar to the content they have been exposed to in a short historical period, which is the so-called user fatigue. Despite the significance for a better user experience, user fatigue is seldom explored by existing recommenders. In fact, there are three main challen… ▽ More Recommender systems filter out information that meets user interests. However, users may be tired of the recommendations that are too similar to the content they have been exposed to in a short historical period, which is the so-called user fatigue. Despite the significance for a better user experience, user fatigue is seldom explored by existing recommenders. In fact, there are three main challenges to be addressed for modeling user fatigue, including what features support it, how it influences user interests, and how its explicit signals are obtained. In this paper, we propose to model user Fatigue in interest learning for sequential Recommendations (FRec). To address the first challenge, based on a multi-interest framework, we connect the target item with historical items and construct an interest-aware similarity matrix as features to support fatigue modeling. Regarding the second challenge, built upon feature cross, we propose a fatigue-enhanced multi-interest fusion to capture long-term interest. In addition, we develop a fatigue-gated recurrent unit for short-term interest learning, with temporal fatigue representations as important inputs for constructing update and reset gates. For the last challenge, we propose a novel sequence augmentation to obtain explicit fatigue signals for contrastive learning. We conduct extensive experiments on real-world datasets, including two public datasets and one large-scale industrial dataset. Experimental results show that FRec can improve AUC and GAUC up to 0.026 and 0.019 compared with state-of-the-art models, respectively. Moreover, large-scale online experiments demonstrate the effectiveness of FRec for fatigue reduction. Our codes are released at https://github.com/tsinghua-fib-lab/SIGIR24-FRec. △ Less

Submitted 22 May, 2024; v1 submitted 19 May, 2024; originally announced May 2024.

Comments: SIGIR 2024

Enhancing Boundary Segmentation for Topological Accuracy with Skeleton-based Methods

Authors: Chuni Liu, Boyuan Ma, Xiaojuan Ban, Yujie Xie, Hao Wang, Weihua Xue, Jingchao Ma, Ke Xu

Abstract: Topological consistency plays a crucial role in the task of boundary segmentation for reticular images, such as cell membrane segmentation in neuron electron microscopic images, grain boundary segmentation in material microscopic images and road segmentation in aerial images. In these fields, topological changes in segmentation results have a serious impact on the downstream tasks, which can even… ▽ More Topological consistency plays a crucial role in the task of boundary segmentation for reticular images, such as cell membrane segmentation in neuron electron microscopic images, grain boundary segmentation in material microscopic images and road segmentation in aerial images. In these fields, topological changes in segmentation results have a serious impact on the downstream tasks, which can even exceed the misalignment of the boundary itself. To enhance the topology accuracy in segmentation results, we propose the Skea-Topo Aware loss, which is a novel loss function that takes into account the shape of each object and topological significance of the pixels. It consists of two components. First, a skeleton-aware weighted loss improves the segmentation accuracy by better modeling the object geometry with skeletons. Second, a boundary rectified term effectively identifies and emphasizes topological critical pixels in the prediction errors using both foreground and background skeletons in the ground truth and predictions. Experiments prove that our method improves topological consistency by up to 7 points in VI compared to 13 state-of-art methods, based on objective and subjective assessments across three different boundary segmentation datasets. The code is available at https://github.com/clovermini/Skea_topo. △ Less

Submitted 7 May, 2024; v1 submitted 29 April, 2024; originally announced April 2024.

MatSAM: Efficient Extraction of Microstructures of Materials via Visual Large Model

Authors: Changtai Li, Xu Han, Chao Yao, Xiaojuan Ban

Abstract: Efficient and accurate extraction of microstructures in micrographs of materials is essential in process optimization and the exploration of structure-property relationships. Deep learning-based image segmentation techniques that rely on manual annotation are laborious and time-consuming and hardly meet the demand for model transferability and generalization on various source images. Segment Anyth… ▽ More Efficient and accurate extraction of microstructures in micrographs of materials is essential in process optimization and the exploration of structure-property relationships. Deep learning-based image segmentation techniques that rely on manual annotation are laborious and time-consuming and hardly meet the demand for model transferability and generalization on various source images. Segment Anything Model (SAM), a large visual model with powerful deep feature representation and zero-shot generalization capabilities, has provided new solutions for image segmentation. In this paper, we propose MatSAM, a general and efficient microstructure extraction solution based on SAM. A simple yet effective point-based prompt generation strategy is designed, grounded on the distribution and shape of microstructures. Specifically, in an unsupervised and training-free way, it adaptively generates prompt points for different microscopy images, fuses the centroid points of the coarsely extracted region of interest (ROI) and native grid points, and integrates corresponding post-processing operations for quantitative characterization of microstructures of materials. For common microstructures including grain boundary and multiple phases, MatSAM achieves superior zero-shot segmentation performance to conventional rule-based methods and is even preferable to supervised learning methods evaluated on 16 microscopy datasets whose micrographs are imaged by the optical microscope (OM) and scanning electron microscope (SEM). Especially, on 4 public datasets, MatSAM shows unexpected competitive segmentation performance against their specialist models. We believe that, without the need for human labeling, MatSAM can significantly reduce the cost of quantitative characterization and statistical analysis of extensive microstructures of materials, and thus accelerate the design of new materials. △ Less

Submitted 2 March, 2024; v1 submitted 10 January, 2024; originally announced January 2024.

Comments: 18 pages, 8 figures, and 5 tables. Updated with revision and code repository

Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics

Authors: Xueyuan Yang, Chao Yao, Xiaojuan Ban

Abstract: Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. Certain methodologies employ sparse Inertial Measurement Units (IMUs) on the human body and harness data-driven strategies to model human poses. However, the reconstruction of motion based solely on sparse IMUs data is inherently fraught with ambiguity, a consequence of numerous identical IMU r… ▽ More Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. Certain methodologies employ sparse Inertial Measurement Units (IMUs) on the human body and harness data-driven strategies to model human poses. However, the reconstruction of motion based solely on sparse IMUs data is inherently fraught with ambiguity, a consequence of numerous identical IMU readings corresponding to different poses. In this paper, we explore the spatial importance of multiple sensors, supervised by text that describes specific actions. Specifically, uncertainty is introduced to derive weighted features for each IMU. We also design a Hierarchical Temporal Transformer (HTT) and apply contrastive learning to achieve precise temporal and feature alignment of sensor data with textual semantics. Experimental results demonstrate our proposed approach achieves significant improvements in multiple metrics compared to existing methods. Notably, with textual supervision, our method not only differentiates between ambiguous actions such as sitting and standing but also produces more precise and natural motion. △ Less

Submitted 26 December, 2023; originally announced January 2024.

Comments: Accepted by AAAI 2024

Dense residual Transformer for image denoising

Authors: Chao Yao, Shuo Jin, Meiqin Liu, Xiaojuan Ban

Abstract: Image denoising is an important low-level computer vision task, which aims to reconstruct a noise-free and high-quality image from a noisy image. With the development of deep learning, convolutional neural network (CNN) has been gradually applied and achieved great success in image denoising, image compression, image enhancement, etc. Recently, Transformer has been a hot technique, which is widely… ▽ More Image denoising is an important low-level computer vision task, which aims to reconstruct a noise-free and high-quality image from a noisy image. With the development of deep learning, convolutional neural network (CNN) has been gradually applied and achieved great success in image denoising, image compression, image enhancement, etc. Recently, Transformer has been a hot technique, which is widely used to tackle computer vision tasks. However, few Transformer-based methods have been proposed for low-level vision tasks. In this paper, we proposed an image denoising network structure based on Transformer, which is named DenSformer. DenSformer consists of three modules, including a preprocessing module, a local-global feature extraction module, and a reconstruction module. Specifically, the local-global feature extraction module consists of several Sformer groups, each of which has several ETransformer layers and a convolution layer, together with a residual connection. These Sformer groups are densely skip-connected to fuse the feature of different layers, and they jointly capture the local and global information from the given noisy images. We conduct our model on comprehensive experiments. Experimental results prove that our DenSformer achieves improvement compared to some state-of-the-art methods, both for the synthetic noise data and real noise data, in the objective and subjective evaluations. △ Less

Submitted 13 May, 2022; originally announced May 2022.

Comments: Updated on 0514

Data privacy protection in microscopic image analysis for material data mining

Authors: Boyuan Ma, Xiang Yin, Xiaojuan Ban, Haiyou Huang, Neng Zhang, Hao Wang, Weihua Xue

Abstract: Recent progress in material data mining has been driven by high-capacity models trained on large datasets. However, collecting experimental data has been extremely costly owing to the amount of human effort and expertise required. Therefore, material researchers are often reluctant to easily disclose their private data, which leads to the problem of data island, and it is difficult to collect a la… ▽ More Recent progress in material data mining has been driven by high-capacity models trained on large datasets. However, collecting experimental data has been extremely costly owing to the amount of human effort and expertise required. Therefore, material researchers are often reluctant to easily disclose their private data, which leads to the problem of data island, and it is difficult to collect a large amount of data to train high-quality models. In this study, a material microstructure image feature extraction algorithm FedTransfer based on data privacy protection is proposed. The core contributions are as follows: 1) the federated learning algorithm is introduced into the polycrystalline microstructure image segmentation task to make full use of different user data to carry out machine learning, break the data island and improve the model generalization ability under the condition of ensuring the privacy and security of user data; 2) A data sharing strategy based on style transfer is proposed. By sharing style information of images that is not urgent for user confidentiality, it can reduce the performance penalty caused by the distribution difference of data among different users. △ Less

Submitted 9 November, 2021; originally announced November 2021.

Comments: 14 pages

Agent-based Simulation Model and Deep Learning Techniques to Evaluate and Predict Transportation Trends around COVID-19

Authors: Ding Wang, Fan Zuo, Jingqin Gao, Yueshuai He, Zilin Bian, Suzana Duran Bernardes, Chaekuk Na, Jingxing Wang, John Petinos, Kaan Ozbay, Joseph Y. J. Chow, Shri Iyer, Hani Nassif, Xuegang Jeff Ban

Abstract: The COVID-19 pandemic has affected travel behaviors and transportation system operations, and cities are grappling with what policies can be effective for a phased reopening shaped by social distancing. This edition of the white paper updates travel trends and highlights an agent-based simulation model's results to predict the impact of proposed phased reopening strategies. It also introduces a re… ▽ More The COVID-19 pandemic has affected travel behaviors and transportation system operations, and cities are grappling with what policies can be effective for a phased reopening shaped by social distancing. This edition of the white paper updates travel trends and highlights an agent-based simulation model's results to predict the impact of proposed phased reopening strategies. It also introduces a real-time video processing method to measure social distancing through cameras on city streets. △ Less

Submitted 23 September, 2020; originally announced October 2020.

End-to-End Learning for Simultaneously Generating Decision Map and Multi-Focus Image Fusion Result

Authors: Boyuan Ma, Xiang Yin, Di Wu, Xiaojuan Ban

Abstract: The general aim of multi-focus image fusion is to gather focused regions of different images to generate a unique all-in-focus fused image. Deep learning based methods become the mainstream of image fusion by virtue of its powerful feature representation ability. However, most of the existing deep learning structures failed to balance fusion quality and end-to-end implementation convenience. End-t… ▽ More The general aim of multi-focus image fusion is to gather focused regions of different images to generate a unique all-in-focus fused image. Deep learning based methods become the mainstream of image fusion by virtue of its powerful feature representation ability. However, most of the existing deep learning structures failed to balance fusion quality and end-to-end implementation convenience. End-to-end decoder design often leads to unrealistic result because of its non-linear mapping mechanism. On the other hand, generating an intermediate decision map achieves better quality for the fused image, but relies on the rectification with empirical post-processing parameter choices. In this work, to handle the requirements of both output image quality and comprehensive simplicity of structure implementation, we propose a cascade network to simultaneously generate decision map and fused result with an end-to-end training procedure. It avoids the dependence on empirical post-processing methods in the inference stage. To improve the fusion quality, we introduce a gradient aware loss function to preserve gradient information in output fused image. In addition, we design a decision calibration strategy to decrease the time consumption in the application of multiple images fusion. Extensive experiments are conducted to compare with 19 different state-of-the-art multi-focus image fusion structures with 6 assessment metrics. The results prove that our designed structure can generally ameliorate the output fused image quality, while implementation efficiency increases over 30\% for multiple images fusion. △ Less

Submitted 23 March, 2021; v1 submitted 17 October, 2020; originally announced October 2020.

Comments: report

Journal ref: NeuroComputing 2021

Turbulent Details Simulation for SPH Fluids via Vorticity Refinement

Authors: Sinuo Liu, Xiaokun Wang, Xiaojuan Ban, Yanrui Xu, Jing Zhou, Jiří Kosinka, Alexandru C. Telea

Abstract: A major issue in Smoothed Particle Hydrodynamics (SPH) approaches is the numerical dissipation during the projection process, especially under coarse discretizations. High-frequency details, such as turbulence and vortices, are smoothed out, leading to unrealistic results. To address this issue, we introduce a Vorticity Refinement (VR) solver for SPH fluids with negligible computational overhead.… ▽ More A major issue in Smoothed Particle Hydrodynamics (SPH) approaches is the numerical dissipation during the projection process, especially under coarse discretizations. High-frequency details, such as turbulence and vortices, are smoothed out, leading to unrealistic results. To address this issue, we introduce a Vorticity Refinement (VR) solver for SPH fluids with negligible computational overhead. In this method, the numerical dissipation of the vorticity field is recovered by the difference between the theoretical and the actual vorticity, so as to enhance turbulence details. Instead of solving the Biot-Savart integrals, a stream function, which is easier and more efficient to solve, is used to relate the vorticity field to the velocity field. We obtain turbulence effects of different intensity levels by changing an adjustable parameter. Since the vorticity field is enhanced according to the curl field, our method can not only amplify existing vortices, but also capture additional turbulence. Our VR solver is straightforward to implement and can be easily integrated into existing SPH methods. △ Less

Submitted 30 September, 2020; originally announced September 2020.

An Interactive Data Visualization and Analytics Tool to Evaluate Mobility and Sociability Trends During COVID-19

Authors: Fan Zuo, Jingxing Wang, Jingqin Gao, Kaan Ozbay, Xuegang Jeff Ban, Yubin Shen, Hong Yang, Shri Iyer

Abstract: The COVID-19 outbreak has dramatically changed travel behavior in affected cities. The C2SMART research team has been investigating the impact of COVID-19 on mobility and sociability. New York City (NYC) and Seattle, two of the cities most affected by COVID-19 in the U.S. were included in our initial study. An all-in-one dashboard with data mining and cloud computing capabilities was developed for… ▽ More The COVID-19 outbreak has dramatically changed travel behavior in affected cities. The C2SMART research team has been investigating the impact of COVID-19 on mobility and sociability. New York City (NYC) and Seattle, two of the cities most affected by COVID-19 in the U.S. were included in our initial study. An all-in-one dashboard with data mining and cloud computing capabilities was developed for interactive data analytics and visualization to facilitate the understanding of the impact of the outbreak and corresponding policies such as social distancing on transportation systems. This platform is updated regularly and continues to evolve with the addition of new data, impact metrics, and visualizations to assist public and decision-makers to make informed decisions. This paper presents the architecture of the COVID related mobility data dashboard and preliminary mobility and sociability metrics for NYC and Seattle. △ Less

Submitted 26 June, 2020; originally announced June 2020.

SESF-Fuse: An Unsupervised Deep Model for Multi-Focus Image Fusion

Authors: Boyuan Ma, Xiaojuan Ban, Haiyou Huang, Yu Zhu

Abstract: In this work, we propose a novel unsupervised deep learning model to address multi-focus image fusion problem. First, we train an encoder-decoder network in unsupervised manner to acquire deep feature of input images. And then we utilize these features and spatial frequency to measure activity level and decision map. Finally, we apply some consistency verification methods to adjust the decision ma… ▽ More In this work, we propose a novel unsupervised deep learning model to address multi-focus image fusion problem. First, we train an encoder-decoder network in unsupervised manner to acquire deep feature of input images. And then we utilize these features and spatial frequency to measure activity level and decision map. Finally, we apply some consistency verification methods to adjust the decision map and draw out fused result. The key point behind of proposed method is that only the objects within the depth-of-field (DOF) have sharp appearance in the photograph while other objects are likely to be blurred. In contrast to previous works, our method analyzes sharp appearance in deep feature instead of original image. Experimental results demonstrate that the proposed method achieves the state-of-art fusion performance compared to existing 16 fusion methods in objective and subjective assessment. △ Less

Submitted 21 August, 2019; v1 submitted 5 August, 2019; originally announced August 2019.

Comments: technological report and fix some errors

WPU-Net: Boundary Learning by Using Weighted Propagation in Convolution Network

Authors: Boyuan Ma, Chuni Liu, Xiaojuan Ban, Hao Wang, Weihua Xue, Haiyou Huang

Abstract: Deep learning has driven a great progress in natural and biological image processing. However, in material science and engineering, there are often some flaws and indistinctions in material microscopic images induced from complex sample preparation, even due to the material itself, hindering the detection of target objects. In this work, we propose WPU-net that redesigns the architecture and weigh… ▽ More Deep learning has driven a great progress in natural and biological image processing. However, in material science and engineering, there are often some flaws and indistinctions in material microscopic images induced from complex sample preparation, even due to the material itself, hindering the detection of target objects. In this work, we propose WPU-net that redesigns the architecture and weighted loss of U-Net, which forces the network to integrate information from adjacent slices and pays more attention to the topology in boundary detection task. Then, the WPU-net is applied into a typical material example, i.e., the grain boundary detection of polycrystalline material. Experiments demonstrate that the proposed method achieves promising performance and outperforms state-of-the-art methods. Besides, we propose a new method for object tracking between adjacent slices, which can effectively reconstruct 3D structure of the whole material. Finally, we present a material microscopic image dataset with the goal of advancing the state-of-the-art in image processing for material science. △ Less

Submitted 30 August, 2019; v1 submitted 22 May, 2019; originally announced May 2019.

Comments: technical report

Journal ref: Journal of Computational Science, 2022

Data augmentation in microscopic images for material data mining

Authors: Boyuan Ma, Xiaoyan Wei, Chuni Liu, Xiaojuan Ban, Haiyou Huang, Hao Wang, Weihua Xue, Stephen Wu, Mingfei Gao, Qing Shen, Adnan Omer Abuassba, Haokai Shen, Yanjing Su

Abstract: Recent progress in material data mining has been driven by high-capacity models trained on large datasets. However, collecting experimental data (real data) has been extremely costly since the amount of human effort and expertise required. Here, we develop a novel transfer learning strategy to address small or insufficient data problem. This strategy realizes the fusion of real and simulated data,… ▽ More Recent progress in material data mining has been driven by high-capacity models trained on large datasets. However, collecting experimental data (real data) has been extremely costly since the amount of human effort and expertise required. Here, we develop a novel transfer learning strategy to address small or insufficient data problem. This strategy realizes the fusion of real and simulated data, and the augmentation of training data in data mining procedure. For a specific task of image segmentation, this strategy can generate synthetic images by fusing physical mechanism of simulated images and "image style" of real images. The result shows that the model trained with the acquired synthetic images and 35% of the real images outperforms the model trained on all real images. As the time required to generate synthetic data is almost negligible, this strategy is able to reduce the time cost of real data preparation by roughly 65%. △ Less

Submitted 28 October, 2019; v1 submitted 12 May, 2019; originally announced May 2019.

Comments: 17 pages, technical report

Journal ref: npj computational materials 2020

Robust and customized methods for real-time hand gesture recognition under object-occlusion

Authors: Zhishuai Han, Xiaojuan Ban, Xiaokun Wang, Di wu

Abstract: Dynamic hand tracking and gesture recognition is a hard task since there are many joints on the fingers and each joint owns many degrees of freedom. Besides, object occlusion is also a thorny issue in finger tracking and posture recognition. Therefore, we propose a robust and customized system for realtime hand tracking and gesture recognition under occlusion environment. First, we model the angle… ▽ More Dynamic hand tracking and gesture recognition is a hard task since there are many joints on the fingers and each joint owns many degrees of freedom. Besides, object occlusion is also a thorny issue in finger tracking and posture recognition. Therefore, we propose a robust and customized system for realtime hand tracking and gesture recognition under occlusion environment. First, we model the angles between hand keypoints and encode their relative coordinate vectors, then we introduce GAN to generate raw discrete sequence dataset. Secondly we propose a time series forecasting method in the prediction of defined hand keypoint location. Finally, we define a sliding window matching method to complete gesture recognition. We analyze 11 kinds of typical gestures and show how to perform gesture recognition with the proposed method. Our work can reach state of the art results and contribute to build a framework to implement customized gesture recognition task. △ Less

Submitted 16 September, 2018; originally announced September 2018.