论文笔记-sign language recognition, translation and production

paper list to read:

• Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

• Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers andMixture Density Networks

• Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives

• Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production

• Ben Saunders, Necati Cihan Camgoz, and Richard Bowden

• Can Everybody Sign Now? Exploring Sign Language Video Generation from 2D Poses

• BABEL: Bodies, Action and Behavior with English Labels, CVPR2021

• Fingerspelling Detection in American Sign Language, CVPR2021

• American Sign Language fingerspelling recognition in the wild. SLT2018

• Fingerspelling recognition in the wild with iterative visual attention. ICCV2019

ESD

Two-stage:

• text to pose sequence
• pose sequence to contiguous sign video

Text2pose

using a Mixture Density Network (MDN):

$\alpha_{i}(x_{1:U})$ is the mixture weight of the $i^{th}$ distribution, regarded as a prior probability of the sign pose being generated from this mixture component. $\phi_{i}(y_t|x_{1:U})$ is the conditional density of the sign pose for the $i^{th}$ mixture.

Similar to auto-regressive vae model.

Pose2video

Fingerspelling Detection

手指拼写的作用：

• 专有名词
• 技术术语
• 缩写等没有对应手势的词汇
• 也用于强调和方便

手指拼写占ASL的 12%-35%. 这个比例比大部分手语词汇量都要大。

Fingerspelling is used for multiple purposes, including for words that do not have their own signs (such as many proper nouns, technical terms, and abbreviations) [39] but also sometimes for emphasis or expediency. Fingerspelling accounts for 12% to 35% of ASL, where it is used more than in other sign languages [40].

手指拼写对应的字母与翻译出来的英语是单调对齐的。这有点类似于翻译中的直接音译。

手指拼写的检测对于下游手语识别任务有显著提升作用。