Robust and Generalizable Sign Language Translation
RoGSiLT aims to advance the translation technologies for German Sign Language (DGS) and French Sign Language (LSF) by overcoming current limitations such as reliance on scarce gloss-annotated data, poor generalization, and unnatural translations. Leveraging a collaboration that integrates expertise in neural machine translation (NMT), speech processing, and computer vision (CV), our approach introduces innovative strategies using self-supervised learning, multimodal neural architectures, and large language models (LLMs). These methods aim to enhance the robustness and generalizability of sign language translation (SLT) systems. We anticipate significant societal impacts, including enhanced accessibility and inclusivity for the Deaf and Hard-of-Hearing (DHH) community, by providing more natural and effective communication tools.
Goals
- Automated methods to automatically align and segment video and text material
- Define a pipeline to parametrically synthesis text-to-SL
- Define a SL-to-text pipeline to robustly recognize SL from video
- Better evaluation metrics for objective and subjective measurements of quality
Research questions
- RQ1 – Can we conceive a “better” representation for sign language?
- RQ2 – Given the scarcity of training material, can we find a strategy to do data augmentation?
- RQ3 – Can we “close the circle” between the two translation directions and perform back-translation?
Partners
- German Research Center for Artificial Intelligence (DFKI)
- SCAAI group, Cognitive Assistants (COS) department, Saarbrücken & Berlin, Germany
- Multilinguality and Language Technology department, Saarbrücken, Germany
- French National Institute for Research in Digital Science and Technology (INRIA)
Funding
This is an DFKI-INRIA joint project proposal funded my the Germany Federal Ministry of Research, Technology and Space (BMFTR), and the French National Institute for Research in Digital Science and Technology.
Duration: April 2026 – March 2029
People
- DFKI
- Department of Cognitive Assistants
- Dr. Fabrizio Nunnari
- Dr. Eleftherios Avramidis
- Department of Multilinguality and Language Technology
- Prof. Josef van Genabith
- Yasser Hamidullah
- Department of Cognitive Assistants
- INRIA
- Multispeech
- Prof. Slim Ouni (Université de Lorraine – Inria)
- Dr. Mostafa Sadeghi (Inria)
- Dr. Sam Bigeard (Inria)
- Multispeech