LT-SignDiff: Long-Tail Video Sign Language Recognition via Diffusion-Synthesized Rare-Class Prototypes

Abstract

Video-based Sign Language Recognition (SLR) is a fundamental accessibility technology, yet real-world sign vocabularies follow an extreme long-tailed distribution, a problem sharpened for low-resource languages such as Vietnamese Sign Language (VSL), where classical imbalance tools (re-weighting, focal loss, GAN-based augmentation) either over-fit the head or leave the tail under-determined. We ask whether the bottleneck in this regime is representation quality or the under-determination of per-class decision boundaries, and whether a generative prior over class prototypes—rather than over raw video—can supply the missing constraint. We propose LT-SignDiff, which repurposes a conditional diffusion model as a generative prior in feature space: it synthesizes a discriminative prototype for each rare class directly in a skeleton encoder's embedding space, conditioned on the k nearest frequent-class prototypes, and injects it into a hybrid cosine classifier. The distinct contribution is that synthesis is performed in prototype space and conditioned on a local semantic neighbourhood—unlike prior diffusion methods that generate pixel-space samples or a single global feature distribution—which can be interpreted as a shrinkage estimator that stabilizes one-shot classifier rows. On a 3,782-class VSL benchmark whose original distribution is extreme (1.60 recordings per class on average, 64% of classes with a single example), LT-SignDiff attains 62.5% Top-1 / 81.4% Top-5 / 58.2% Macro-F1, improving over the strongest non-generative baseline by +13.3 Top-1 / +12.4 Macro-F1 and over the best prior VSL baseline by +32.7 Top-1. Ablations attribute the majority of the tail-class gain to the diffusion prototype synthesis, indicating that for low-resource SLR the most effective place to be generative is the prototype space, not the pixel space.