YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus

We retrieved candidate videos in a similar way to YouTube-ASL [30], by fetching videos tagged with Knowledge Graph entities related to sign language generally or any individual sign language with its own tag as of July 2023.²²2We refreshed the dataset in May 2024 to include newer videos from accepted channels, with some light manual review. As with YouTube-ASL, the recall of these tags is limited in that they are not aware of sign language in the video content itself (i.e., they do not use sign language detection and sign language identification classifiers), so they may miss videos or channels that use sign language but do not explicitly mention it. The pool of videos that we fetched from is also restricted in some generic ways, such as that the videos must be public and listed.

We applied the same filtering steps as YouTube-ASL: select only videos with manually uploaded captions, and remove videos with duration $<$10 seconds or $>$5 hours, width $<$480 pixels or height $360$ pixels, and frame rate $<$15fps or $>$60fps. We added one additional step: remove videos where captions cover $<$40% of the duration.

Unlike YouTube-ASL, we did not exclude videos of conversations with more than one signer in frame at a time. While our pose-based baselines only support one person’s input, we feel that there is no reason to exclude these videos from the corpus itself. We are not aware of any prior works that study sign language translation from videos with multiple signers; some datasets like the Public DGS Corpus [8] consist of conversational data, but they are recorded with one camera per signer. Being able to handle this kind of data is important both in the near term to make the most of the limited resources available and in the long term to support multi-signer applications.

The result of this process was a list of 81,623 candidate videos that might contain signed content with high-quality captions.

YouTube-ASL used 3 native ASL signers as annotators to identify which of the individual candidate videos contained ASL and had high-quality, well-aligned English captions, labelling videos over the course of several months and several rounds of iteration on the annotation tool/labelling standards. This kind of annotation does not scale well to massively multilingual data because it is difficult and expensive to onboard native (or even just proficient) signers to annotate videos in tens of (extremely low resource) sign languages. Instead, we follow Kreutzer et al. [12]’s observation that many data filtering tasks can be done even without full understanding of the content. Therefore, the first author (a non-native hearing signer with experience in several sign languages) served as the annotator, triaging the corpus over the course of a four-day weekend using a combination of per-channel annotations and targeted audits. The triage proceeded as follows:

First, we automatically included the YouTube-ASL videos in our corpus and removed these from the manual review. Then we grouped videos by channel and sorted by total duration in descending order, with random ordering for videos within each channel. We used heuristic classifiers based on public text metadata for each video and channel (e.g., title, description, caption language) to provisionally label each video with a sign language. For each channel, the author sampled several random videos and judged whether they matched the acceptance criteria (described below), then corrected the sign language label if relevant. If all the sampled videos were good, the channel was tentatively accepted; if the videos were bad on balance, it was rejected; if it was mixed but promising, the author investigated further and was able to exclude individual bad videos. The author triaged the first 1000 channels of 18000 (down to about 1 hour of content per channel), then continued into the 3000-6000 range, skipping videos predicted to be in ASL in order to prioritize the long tail of sign languages.³³3An observation: the acceptance rates for videos in different sign languages were dramatically different. For example, the vast majority of content from Central & South America was live-interpreted events with small interpreters and poor caption alignment. We expect that high-quality produced content is more common in high-income countries, so it will be important for future work to leverage lower quality data in order to be more globally inclusive. Finally, we sorted the videos for each channel by duration in descending order and the first author made a second pass, checking outlier videos with unusually long durations for their channels⁴⁴4Sometimes channels with high-quality produced content with relatively consistent duration $\sim$10 minutes would have long live-interpreted events with different translation and caption quality. to reduce heavy-hitter errors.

Now, a description of our annotation standard for individual videos and some shallow signals to recognize these features even without full understanding:

• •

  The video should feature sign language. While there is such a thing as fake sign language interpreters, they do not typically create YouTube channels full of themselves gesturing nonsensically, so there is not much adversarial content to sift through. The closest thing to this is inexperienced sign language learners, especially producing loose translations of music.⁵⁵5We accept learners as long as they seem relatively competent. Models should be able to understand less proficient signers, especially given that many deaf people are not exposed to sign language in childhood and reach varying levels of proficiency [16]. Less proficient signing can be easily distinguished from native signing, so it should not harm the quality of sign language understanding (especially if the data is only used for pretraining), but more care is required for generation. An annotator proficient in one sign language can generally distinguish these cases (and others like interpretive dance/gesture) even for other sign languages.

• •

  The signer(s) should be the primary content. If a signer is interpreting spoken content, the video of the speaker (if present) should be significantly smaller than the signer. If the person is switching off between signing and speaking, or a speaker is the primary content for a significant fraction of the video, it should be rejected.

• •

  The video should have relatively well-aligned captions. There are a few signals you can use for this: if the video has no speech track, the captions are virtually always well-aligned;⁶⁶6These are typically channels with natively Deaf-produced content. you can validate this by seeing whether the changes in captions occur at boundaries in sign language phrases (which are marked pretty universally with pauses and nonmanual signals like head tilt or eyebrow movement). If there is a speech track, you can listen to whether the caption timing matches the speech track. We accepted videos where the captions were aligned to the speech track as long as they were not off by more than a couple of seconds; usually this corresponds to an after-the-fact voiceover translation, not live interpretation.

• •

  Label which sign language the video is in. It was almost always possible to classify the sign language with high confidence based on the video title/description and caption language, or at worst the channel’s description (including its stated country). The failures in automatic classification mostly came from International Sign, often specified as just “IS”, which was not automatically tagged. In the rare cases where we could not determine the language with high confidence, we marked the sign language as unknown.

In order to sanity check the end-to-end quality of this triage process, the first author ran two additional audits on the final dataset (excluding the videos from YouTube-ASL, since we were already confident in their quality). The author rated two samples of 100 videos, one sampled uniformly per video id, and the other with videos sampled with probability proportional to their video duration. The author loaded each video and scrubbed through the frame previews across time to identify cuts where content and captioning habits may change.

In the sample with random videos by ID, we found: 1 video that strictly should not have been included in the dataset (a brief video with captioned narration), 1 borderline video with a waist-up interpreter next to a video of a speaker’s face only, and 3 videos with sign-along versions of songs that used appropriate vocabulary but didn’t really provide a complete translation.

In the sample proportional to video duration, we found: 0.76% of content that was speech rather than signing (from a channel with otherwise entirely signed content), and 0.96% that consisted of interstitials mixed within otherwise good content, where the interstitials had signing but no captions.

We consider these error rates acceptable.

The final human-triaged YouTube-SL-25 corpus consists of 39197 videos totaling 3207 hours of content (covered by 2980 hours of captions = 2.16M captions = 104M characters)⁷⁷7We used classifiers to evaluate whether the captions were toxic, hateful, violent, and sexual with confidence ¿0.8, which flagged ¡1% of captions. We manually audited the captions flagged with highest confidence and none merited exclusion from the dataset: the “toxic” and “hateful” content were false positives, the “violent” content was news about violent events, and the “sexual” content was news or educational content about sex. across $>$25 sign languages.mom Like in YouTube-ASL, we approximately lower bound the number of unique signers using the number of unique channels, 3072, and expect that this is quite an underestimate as channels (or even individual videos) featuring multiple signers are common. This is only somewhat larger than YouTube-ASL’s 2519 channels, which is the main downside of our triage-based annotation approach: it captures the bulk of the hours but not the long tail of unique channels.

As discussed in Uthus et al. [30]’s related work section, to the best of our knowledge YouTube-ASL and now YouTube-SL-25 are the only sign language translation datasets that aggregate content from a wide variety of automatically mined channels, as opposed to others which either pay participants to create new data [3, 31, 11, 8] or curate content from a small number of content creators [27, 1, 26, 18, 6, 10, 15]. This has implications for diversity in topics/content and video production style.⁸⁸8Note that even though YouTube-SL-25 is open-domain overall, there may be discretization effects in the long tail of languages, where a language is represented by only a few channels and therefore in effect is closed-domain.

See Figure 1 for a depiction of the relative weight of each language on a world map and Table 1 for a breakdown of hours by language, comparing to the largest prior parallel dataset for each language (both open- and closed-domain). Note that prior works have studied sign languages with no representation in YouTube-SL-25, like AfriSign [7] & JWSign [6] (primarily languages in Central & South America and Africa), and CSL-Daily [35] (China); these languages aren’t included in the table. The distribution of language data can be attributed to population size, economic development, and political factors. Other supplementary sources of data, such as Bible translations [7, 6] and national interpreted broadcasts [1, 2], could provide more balanced data distributions.

See Figure 2 for an estimated demographic breakdown of YouTube-SL-25 by skin tone and perceived gender presentation, predicted by automated classifiers. Dark skin is globally underrepresented in the dataset, with only 1.9% (60 hours) of the data at Monk 6-10. Some of this can be explained by overrepresentation of countries in the Global North (and the only two countries in the Global South with a significant fraction of the dataset, Brazil and India, have substantial variation in skin tone). Ensuring demographic fairness remains an important problem for future work.

We build on the unified modeling approach adopted by the FLEURS-ASL baselines [29], which are an evolution of the YouTube-ASL baselines [30]. In brief, this approach finetunes a pretrained encoder-decoder language model (T5-v1.1 Small) to take as input 256 tokens of text (control tokens and caption context) and 512 frames of MediaPipe Holistic landmarks at half frame rate (corresponding to a random video span), and output 256 tokens of text (including timestamps).

The only modification we make is to extend the mixture to support multiple source and target languages, as well as the sign language identification task with probability 0.1 (where it classifies the language and then proceeds to translate the input). Like in FLEURS-ASL, we train first on caption-level clips and then an equal mixture of caption-level and random clip-level data. See Figure 3 for a depiction of the modifications to the control token format.⁹⁹9We tried to change the pretrained model from T5-v1.1 Small [21] to mT5 Small [32] so that languages besides English could benefit from pretraining and better tokenization, but in initial experiments mT5 took about 1/3 more steps to converge and achieved worse results. We therefore ran the full set of experiments with T5 only.

We compare three models: T5 with only text pretraining (i.e., the original pretrained T5 Small), T5 with continued training on YouTube-SL-25’s 1400-hour ASL subset, and T5 with continued training on the full YouTube-SL-25 dataset. See Appendix A for training details.

See FLEURS-ASL [29] for discussion of how their baseline approach (in particular, training on random clips and caption tracks rather than tightly clipped individual captions) relates to prior modeling work. Insofar as YouTube-SL-25 is slightly noisier than YouTube-ASL, we slightly push the boundaries in terms of training on weakly aligned data. With respect to our modifications: modeling multiple sign languages in one model is not new [33, 34, 7, 6]. We are not aware of any prior works that train sign language identification as a task within translation models (the few prior works on sign language identification [4, 5] use low-level phonetic features only), but this is not especially novel given its use in Whisper for speech input [20].

We provide sentence-level translation (scored with BLEURT [24]) and sign language identification (scored with top-1 accuracy) results on benchmarks for 4 sign languages: American Sign Language (ase, FLEURS-ASL [29] & How2Sign [3]), Swiss German Sign Language (sgg, WMT23 SignSuisse [15]), Swiss French Sign Language (ssr, WMT23 SignSuisse [15]), and Swiss Italian Sign Language (slf, WMT23 SignSuisse [15]). We provide both zero-shot scores and finetuned scores where relevant.¹⁰¹⁰10For translation, the model is separately finetuned for each dataset, checkpoint selected based on BLEU on the validation set. For sign language identification, zero-shot scores mean that the model is briefly finetuned on YouTube-SL-25 rebalanced to the 4 sign languages with equal weight, and finetuned scores mean that the model is finetuned on an equally weighted mixture of the benchmarks’ training sets. We don’t finetune on FLEURS-ASL, so the finetuned langid scores are after finetuning on How2Sign.

Table 2 shows quantitative results on both tasks, and Table 3 shows a sample of qualitative translation examples. We see that on both translation and lang id, sign language pretraining is substantially better than none (as in Uthus et al. [30]) and multilingual transfer helps both the higher-resource sign language (ASL) and the lower-resource sign languages within YouTube-SL-25.¹¹¹¹11Some prior works [7, 6] did not see strict improvement from modeling additional sign languages; we expect that this was a model capacity issue (for pretraining, or finetuning on multiple languages simultaneously rather than one at a time). As expected, finetuning gives large gains—especially when the language is poorly represented in YouTube-SL-25—and benefits from multilingual pretraining. For example, our finetuned sgg result of 37.7 BLEURT (7.5 BLEU) far exceeds the top WMT23 score of 23.6 BLEURT (0.3 BLEU).

Note that for sign language identification, finetuning is somewhat ill-conceived because some datasets have train/test signer overlap or just generally unique recording conditions that could be used a shortcut. For example, langid quality on FLEURS-ASL decreases when pretrained on ASL data and finetuned on How2Sign, even compared to no sign language pretraining—with accuracy varying dramatically across the 3 signers within the test set, averaging 18.3% vs. 94.9% vs. 77.0%. This suggests that different kinds of pretraining may cause different features to be surfaced during finetuning, which may overfit or not transfer with respect to broader domains. Our zero-shot results are nontrivial despite the severe class imbalance. A proper sign language identification benchmark needs more consistency across languages; we were unable to use SP-10 [33] for licensing reasons.