论文笔记-sign language recognition, translation and production

paper list to read:

  • Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

  • Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers andMixture Density Networks

  • Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives

  • Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production

  • Ben Saunders, Necati Cihan Camgoz, and Richard Bowden

  • Can Everybody Sign Now? Exploring Sign Language Video Generation from 2D Poses

  • BABEL: Bodies, Action and Behavior with English Labels, CVPR2021

  • Fingerspelling Detection in American Sign Language, CVPR2021

  • American Sign Language fingerspelling recognition in the wild. SLT2018

  • Fingerspelling recognition in the wild with iterative visual attention. ICCV2019

ESD

Two-stage:

  • text to pose sequence
  • pose sequence to contiguous sign video
image-20220105113847936

Text2pose

using a Mixture Density Network (MDN):

image-20220105113232986

$\alpha_{i}(x_{1:U})$ is the mixture weight of the $i^{th}$ distribution, regarded as a prior probability of the sign pose being generated from this mixture component. $\phi_{i}(y_t|x_{1:U})$ is the conditional density of the sign pose for the $i^{th}$ mixture.

Similar to auto-regressive vae model.

Pose2video

Fingerspelling Detection

手指拼写的作用:

  • 专有名词
  • 技术术语
  • 缩写等没有对应手势的词汇
  • 也用于强调和方便

手指拼写占ASL的 12%-35%. 这个比例比大部分手语词汇量都要大。

Fingerspelling is used for multiple purposes, including for words that do not have their own signs (such as many proper nouns, technical terms, and abbreviations) [39] but also sometimes for emphasis or expediency. Fingerspelling accounts for 12% to 35% of ASL, where it is used more than in other sign languages [40].

手指拼写对应的字母与翻译出来的英语是单调对齐的。这有点类似于翻译中的直接音译。

手指拼写的检测对于下游手语识别任务有显著提升作用。

论文笔记-sign language recognition, translation and production

http://www.panxiaoxie.cn/2021/07/11/论文笔记-sign-language-recognition-and-translation/

作者

Xie Pan

发布于

2021-07-11

更新于

2022-01-05

许可协议

评论