SignSpeak: Open-Source Time Series Classification for ASL Translation

American Sign Language (ASL) is the most prominent sign language in North America [1], yet as of 2021, only 0.15% of Americans are fluent in it [2]. This low figure causes significant challenges for hearing and speech-impaired individuals, including limited access to education, opportunities and essential services, leading to isolation and depression[3].

To address these barriers, prior work using optical-based methods has shown strong results in translating images of ASL gestures to speech; however, they are limited in real-world applicability[4, 5]. CNN and vision-based transformer models necessitate using a camera pointed at a user’s hands while signing, which is impractical in many contexts. Additionally, the use of cameras also presents a privacy risk by capturing the user and surrounding individuals while requiring considerable computing resources as frames must be sent to a server. This is infeasible and limits the scope of optical-based ASL translation within a real-world context.

Sensor-based models using embedded devices have been introduced to treat ASL as a time-series multi-label classification problem to address the limitations of optical systems. However, many of these datasets are private [6, 7] and have not been trained on a well-practiced sign-based language such as ASL [8], limiting their applicability. To address this, we introduce SignSpeak, an open-source ASL dataset comprising of 7200 recordings of 36 classes. Our dataset was recorded using five low-cost flex sensors, one for each finger, for all letters and numbers in ASL. The scale of our dataset enables researchers to test novel models on a dataset collected in an environment aimed at real-world feasibility to progress ASL-to-speech efforts. We extensively benchmarked various methods on this dataset, with our best result achieving 92% categorical accuracy, which matches or exceeds previous ASL time-series classification work [9].¹¹1 The GitHub codebase and dataset are available at https://github.com/adityamakkar000/ASL-Sign-Research and https://doi.org/10.7910/DVN/ODY7GH respectively.

Previous work using a glove-based apparatus involves sensory devices such as flex sensors and inertial measurement units (IMUs). Amin et al. [6] utilize flex sensor gloves to capture 37 hand gestures (numbers 0-10 and letters A-Z) using MLPs achieving 97.6% accuracy. However, a fundamental flaw limits real-world applicability as the measurements are static and recorded at only one point during the gesture. This fails to account for ASL’s dynamic nature since each sign is a sequence of motions that must be continuously measured. Furthermore, the dataset is closed-source, prohibiting others from building on it.

Lee et al. [7] developed a glove taking continuous measurements of 6 inertial measurement units, including an accelerometer, gyroscope, and magnetometer. They report a 99.87% accuracy; however, this study presents a drawback: each input is 10-15 seconds long and impractical for real-world signing, which is performed at a significantly faster rate of 4 syllables per second [10]. Similar to the previous work, the dataset was not released publicly.

Tan et al. [5] developed a 28-sensor glove which recorded 63 data channels to train an LSTM model.¹¹1Certain sensors measured multiple data channels. See the referenced paper for more details. Multiple data channels and sensory equipment significantly increase the glove’s cost, decreasing its affordability. In addition, less sensory equipment can produce similar results in commercial use. Králik and Šuppa [8] utilized a transformer architecture to achieve over 99% accuracy on a synthetic glove-collected gesture dataset, preventing its applicability to the ASL community.

We differ from previous work by introducing an open-source ASL dataset measuring 5 flex sensor channels. It includes 200 samples for all alphanumeric classes, allowing for a cost-effective and resource-efficient glove with broad applicability for the ASL community.

All models were trained with AdamW [18], with $B_{1}=0.9,B_{2}=0.999$, a weight decay of 0.01, and a plateau learning rate decay on validation loss with a patience of 20 epochs of 0.5 starting from 0.001 until a minimum learning rate of 0.0001. RNNs were trained with a batch size of 64 for 15 minutes on an M2, and the Encoder was trained with a batch size of 256 for 15 minutes on a T4 GPU. All models used a 0.2 dropout probability. The metrics used to evaluate all models were categorical accuracy and the F1-Score. We utilized a stratified 5-fold validation and reported the standard deviation and average result of the held-out folds.

The results in Table 1 indicate methods on private datasets do not generalize to the SignSpeak dataset. This may be due to a reduction in the number of data channels. This can be seen with a model such as the Transformer where a lack of data channels reduces its performance. In particular, we found that simple models such as a stacked GRU perform the best, whereas models such as a dense LSTM proposed by Toro-Ossaba et al. [9] do not achieve near state-of-the-art results. We believe that the potential for Transformer-based architectures can be unlocked with more training data, which can then be further fine-tuned on the SignSpeak dataset. Our leading RNN and Encoder models still maintain above 99.5% traditional accuracy, demonstrating performance on par with previous studies.

Additionally, the Transformer architecture has the largest difference between F1-score and accuracy, indicating a bias towards certain classes. Figure 2 displays the confusion matrix and it is evident the low accuracy is due to specific classes such as ’E’ and ’L’. Specifically, the Encoder incorrectly predicts ’L’ 36% of the time when the actual label is ’E’. Additionally in ASL, these letters do not share the same features; ’E’ is very similar to a letter such as ’A’. This indicates the model’s over-predicts between certain classes and could be outlier patterns in the dataset. Analyzing this class with stacked GRU and LSTM models, they predict ’L’ instead of ’E’ 16% and 13% of the time, respectively. This indicates it is a learned bias of an Encoder model but over-fitting is still present in all models.

The models presented in this study only required a moderate amount of computing power to achieve 92% accuracy. In the future, leveraging more powerful computing resources can enable the implementation of larger-scale architectures to further enhance performance. Additionally, the gestures chosen in this dataset (alphanumeric classes) reflect an extremely limited subsection of real-world ASL; thus, future work is aimed at expanding the dataset by collecting data and creating classes for phrases and actions that resemble daily communication making the product viable for commercial use. Lastly, while our measurements were recorded at 36 Hz, which is slower than average ASL signing rates, we anticipate that using an improved MCU will allow us to increase this frequency to 200 Hz, aligning with more realistic signing speeds [10]. These advancements will expand on our existing research and contribute to a more refined product that can facilitate the integration of hearing and speech-impaired individuals into society.

In this study, we presented SignSpeak, an open-source dataset collected using a custom low-cost glove architecture benchmarked on time-series classification models to mimic real-time ASL translation. We found that a stacked GRU achieves the strongest results on categorical accuracy. SignSpeak benefits speech and hearing-impaired communities by providing a way to benchmark models on a universal dataset. Our work on Signspeak can provide a foundation for researchers to build upon our open-source dataset, leveraging supervised learning techniques to deliver assistive and accessible technology to communities in need.