neural-based methods aim to select potential parallel sentences from monolingual corpora in the same domain. However, these neural models need to be trained on a large parallel dataset first, which is not applicable to language pairs with limited supervision.
通过平行语料训练翻译模型，进而从单语中选择 domain related sentences. 这并不是完全的无监督，还是需要有限的平行语料得到 NMT 模型之后，去选择合适的单语。
Parallel sentence extraction from comparable corpora with neural network features, LERC 2016
Bilingual word embeddings with bucketed cnn for parallel sentence extraction, ACL 2017
Extracting parallel sentences with bidirectional recurrent neural networks to improve machine translation, COLING 2018
完全的无监督机器翻译
The main technical protocol of these approaches can be summarized as three steps:
Initialization
Language Modeling
Back-Translation
Initialization
Given the ill-posed nature of the unsupervised NMT task, a suitable initialization method can help model the natural priors over the mapping of two language spaces we expect to reach.
初始化的目的基于自然语言的一些先验知识来对两种语言的映射关系进行建模。
there two main initiazation methods:
bilingual dictionary inference 基于双语词典的推理
Word translation without parallel data. Conneau, et al. ICLR 2018
本文作者采用的是 Conneau, et al. 中的方式，并且类似于 Lample 2018b 中的方式两种语言共享 bpe(需要在看下相关论文). 这里实际上就是训练得到两种语言的 word embedding，并不是 word2vec 那种对单种语言的无监督，而是训练得到两种语言的 share embedding.
language modeling
Train language models on both source and target languages. These models express a data-driven prior about the composition of sentences in each language.
Improving neural machine translation models with monolingual data. ACL 2016
Extract-Edit
Extract: 先根据前两步得到的 sentence 表示，从 target language space 中选择与 source sentence 最接近的 sentence（依据相似度？）.
Edit: 然后对选择的 sentence 进行 edit.
作者还提出了一个 comparative translation loss。
Extract
因为在 language model 阶段作者已经共享了 encoder 和 decoder，所以在这个场景下对于 two language 的表示，都可以用 encoder 得到。
在 target language space 中选择出与 source sentence 最接近的 top-k extracted sentences. 为什么是 top-k 而不是 top-1 呢，确保召回率，并获得更多更相关的 samples.
Edit
简单点就是 max-pooling + decode
employ a maxpooling layer to reserve the more significant features between the source sentence embedding $e_s$ and the extracted sentence embedding $e_t$ ($t\in M$), and then decode it into a new sentence $t’$.
Basically, the translation model is trying to minimize the relative distance of the translated sentence t* to the source sentence s compared to the top-k extracted-and-edited sentences in the target language space. Intuitively, we view the top-k extracted-and-edited sentences as the anchor points to locate a probable region in the target language space, and iteratively improve the source-to-target mapping via the comparative learning scheme.
Adversarial Objective
we can view our translation system as a “generator” that learns to generate a good translation with a higher similarity score than the extracted-and-edited sentences, and the evaluation network R as a “discriminator” that learns to rank the extracted- and-edited sentences (real sentences in the target language space) higher than the translated sentences.
[123] Context-aware nat- ural language generation for spoken dialogue systems. Zhou et al, 2016 COLING
adopted an encoder-decoder LSTM-based structure to incorporate the question information, semantic slot values, and dialogue act type to generate correct answers. It used the attention mechanism to attend to the key information conditioned on the current decoding state of the decoder. Encoding the di- alogue act type embedding, the neural network-based model is able to generate variant answers in response to different act types.
Key-value retrieval networks for task-oriented dialogue, 2017 augmented existing recurrent network architectures with a differentiable attention-based key-value retrieval mechanism over the entries of a knowledge base, which is inspired by key-value memory networks.
A challenging problem in current sequence-to-sequence dialogue systems is that they tend to generate trivial or non-committal, universally relevant responses with little meaning, which are often involving high frequency phrases along the lines of I dont know or Im OK.
A conditional variational framework for dialog generation. ACL 2017 conditioned the latent variable on explicit attributes to make the latent variable more interpretable. These attributes can be either manually assigned or automatically detected such topics, and personality.
enhancing encoder network to obtain high quality context vector
representation of source sentence.
Large scale target-side data:
boosting fluency for machine translation when decoding.
The methods of using monolingual data
Multi-task learning
Target-side language model: Integrating Language Model into the Decoder
shallow fusion
both an NMT model (on parallel corpora) as well as a recurrent neural network language model (RNNLM, on larger monolingual corpora) have been pre-trained separately before being integrated.
Shallow fusion: rescore the probability of the candidate words.
deep fusion
multi-task learning
Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning, EMNLP, 2017
the network ‘unlearns’ its conditioning on the source context if the ratio of monolingual training instances is too high.
Improving Neural Machine Translation Models with Monolingual Data, Sennrich et al, ACL 2016
Self-learning
Synthetic target sentences from source-side mono-data:
Build a baseline machine translation (MT) system on parallel data
Translate source-side mono-data into target sentences
Real parallel data + pseudo parallel data
reference
Improving Neural Machine Translation Models with Monolingual Data, Sennrich et al, ACL 2016
Using Monolingual Data in Neural Machine Translation: a Systematic Study, Burlot et al. ACL 2018
Copied Monolingual Data Improves Low-Resource Neural Machine Translation, Currey et al. 2017 In Proceedings of the Second Conference on Machine Translation
Semi-Supervised Learning for Neural Machine Translation, Cheng et al. ACL 2016
Exploiting Source-side Monolingual Data in Neural Machine Translation, Zhang et al. EMNLP 2016
Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning, Domhan et al. EMNLP 2018
On Using Monolingual Corpora in Neural Machine Translation, Gulcehre, 2015
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation, EMNLP 2018
Understanding Back-Translation at Scale, Edunov et al. EMNLP 2018
r-dim self-attention：有 r 个上述的 attention vector，并转换成矩阵形式，$A\in R^{n\times r}$ 与 encode 之后的句子表示 H 加权求和得到 embedding matrix $M\in R^{r\times 2u}$
$$A=softmax(W_{s2}tanh(W_{s1}H^T))$$
$$M=AH$$
penalization term
上面模型中使用 r 个 attention，很可能会出现冗余的情况，也就是得到的 r 个 attention vector( 论文中说的是 summation weight vectors) 可能得到的是同一个东西，所以需要 diversity.
The best way to evaluate the diversity is definitely the Kullback Leibler divergence between any 2 of the summation weight vectors. However, we found that not very stable in our case. We conjecture it is because we are maximizing a set of KL divergence (instead of minimizing only one, which is the usual case), we are optimizing the annotation matrix A to have a lot of sufficiently small or even zero values at different softmax output units, and these vast amount of zeros is making the training unstable. There is another feature that KL doesn’t provide but we want, which is, we want each individual row to focus on a single aspect of semantics, so we want the probability mass in the annotation softmax output to be more focused. but with KL penalty we cant encourage that.
对于自然语言的语义表示，需要的大量的 common sense 和 world knowledge. 前人的研究使用统计的方法，例如 WordNet，仅仅只利用了有限的词典知识（lexicographic knowledge），并不能有效的利用语言本身背后的背景知识（ background knowledge）。
Our method represents meaning in a high-dimensional space of concepts derived from Wikipedia, the largest encyclopedia in existence. We explicitly represent the meaning of any text in terms of Wikipedia-based concepts.
显示的使用 wiki 中的概念（concepts）来表示任意长度的 text.
作者通过文本分类和计算自然语言的文本片段之间的相似度来验证 ESA 的有效性。由于语义表示使用的是 natural concepts，ESA 模型的可解释性非常强。
传统的方法：
词袋模型: 将 text 看作是 unordered bags of words, 每一个单词看作是一维特征。但是这并不能解决 NLP 中的两个主要问题： 一词多义和同义词（polysemy and synonymy）。
隐语义分析：Latent Semantic Analysis (LSA)
LSA is a purely statistical technique, which leverages word co-occurrence information from a large unlabeled corpus of text. LSA does not use any explicit human-organized knowledge; rather, it “learns” its representation by applying Singular Value Decomposition (SVD) to the words-by-documents co-occurrence matrix. LSA is essentially a dimensionality reduction technique that identifies a number of most prominent dimensions in the data, which are assumed to correspond to “latent concepts”. Meanings of words and documents are then represented in the space defined by these concepts.
However, lexical resources offer little information about the different word senses, thus making word sense disambiguation nearly impossible to achieve.Another drawback of such approaches is that creation of lexical resources requires lexicographic expertise as well as a lot of time and effort, and consequently such resources cover only a small fragment of the language lexicon. Specifically, such resources contain few proper names, neologisms, slang, and domain-specific technical terms. Furthermore, these resources have strong lexical orientation in that they predominantly contain information about individual words, but little world knowledge in general.
Observe that an encyclopedia consists of a large collection of articles, each of which provides a comprehensive exposition focused on a single topic. Thus, we view an encyclopedia as a collection of concepts (corresponding to articles), each accompanied with a large body of text (the article contents).
Ben Bernanke, Federal Reserve, Chairman of the Federal Reserve, Alan Greenspan (Bernanke’s predecessor), Monetarism (an economic theory of money supply and central banking), inflation and deflation.
对于文本：”Apple patents a Tablet Mac”
相关的 concept: Apple Computer 2 , Mac OS (the Macintosh operating system) Laptop (the general name for portable computers, of which Tablet Mac is a specific example), Aqua (the GUI of Mac OS X), iPod (another prominent product by Apple), and Apple Newton (the name of Apple’s early personal digital assistant).
ESA 对一个 texts 的表示是 wiki 中所有的 concept 的 weighted combination，这里为了展示方便，只列举了最相关的一些 concept.
ESA(explicit semantic analysis)
通过 wiki 得到一个 basic concepts: $C_1, C_2,…, C_n$, 其中 $C_k$ 都是来源于 wiki. 表示一个通用的 n 维语义空间。
然后将任意长度的文本 t 表示成与上述向量长度相同的 权重向量 $w_1, w_2,…, w_n$ 分别表示 t 与 $C_k$ 之间的相关程度。
接下来两个步骤就是：
the set of basic concepts
the algorithm that maps text fragments into interpretation vectors
如何构建 concept 集合
1.using Wikipedia as a Repository of Basic Concepts
而 Hiton 提出了 Capsule 就具有很好的可解释性，那么其中的 “抛弃梯度下降” 又是怎么一回事呢？苏神在 blog 中给了很好的解释。
胶囊的计算
在前面的 blog 中我们已经理解了什么是“胶囊”。神经元是标量，胶囊就是向量！Hinton的理解是：每一个胶囊表示一个属性，而胶囊的向量则表示这个属性的“标架”。也就是说，我们以前只是用一个标量表示有没有这个特征（比如有没有羽毛），现在我们用一个向量来表示，不仅仅表示有没有，还表示“有什么样的”（比如有什么颜色、什么纹理的羽毛），如果这样理解，就是说在对单个特征的表达上更丰富了。不仅如此，上一篇 blog 中有提到的 CNN 中的不足，主要在于两点，一是 max pooling 丢失了部分信息（这在低层次的layer中可能影响不大，但是在高层次的layer就会有比较大的影响），二是 CNN 不能提取低维特征与高维特征之间在空间中的相对位置信息。而胶囊的方向能表示这一部分信息。比如下面这张图就很明显的表示出来了。
然后反复迭代。说白了，输出是输入的聚类结果，而聚类通常都需要迭代算法，这个迭代算法就称为“动态路由”。至于这个动态路由的细节，其实是不固定的，取决于聚类的算法，比如关于Capsule的新文章《MATRIX CAPSULES WITH EM ROUTING》就使用了Gaussian Mixture Model来聚类。
Internal data representation of a convolutional neural network does not take into account important spatial hierarchies between simple and complex objects.
Inspired by this idea, Hinton argues that brains, in fact, do the opposite of rendering. He calls it inverse graphics: from visual information received by eyes, they deconstruct a hierarchical representation of the world around us and try to match it with already learned patterns and relationships stored in the brain. This is how recognition happens. And the key idea is that representation of objects in the brain does not depend on view angle.
So at this point the question is: how do we model these hierarchical relationships inside of a neural network?
那么我们如何通过神经网络对一个层次信息进行建模呢？
Capsules
Capsules introduce a new building block that can be used in deep learning to better model hierarchical relationships inside of internal knowledge representation of a neural network.
例如，如果低层次capsule的prediction vector远离capsule J中“correct”预测的红色集群，并且接近capsule K 中的“true”预测的红色集群，那么低层次capsule就会调高capsule K对应的权重，降低capsule J对应的权重。最后，如果高层次capsule接收到的这些prediction都agree这个capsule，那么这个capsule就会被激活，处于active状态，这就叫Routing by agreement。
Lower level capsule will send its input to the higher level capsule that “agrees” with its input. This is the essence of the dynamic routing algorithm.
Recent approaches that concatenate embeddings derived from other tasks with the input at different layers (Peters et al., 2017; McCann et al., 2017; Peters et al., 2018) still train the main task model from scratch and treat pretrained embeddings as fixed parameters, limiting their usefulness.
这篇 paper 是在 elmo 之后，而 elmo 虽然相对出名，影响力更大，但是 elmo 仍旧只是一种 word embedding 的预训练，在下游任务中还是需要从头训练模型。
ELMo有以下几个步骤：
利用LM任务进行预训练
再利用目标领域的语料对LM模型做微调
最后针对目标任务进行 concatenate embedding，然后训练模型
pretraining LM:
In light of the benefits of pretraining (Erhan et al., 2010), we should be able to do better than randomly initializing the remaining parameters of our models. However, inductive transfer via finetuning has been unsuccessful for NLP (Mou et al., 2016). Dai and Le (2015) first proposed finetuning a language model (LM) but require millions of in-domain documents to achieve good performance, which severely limits its applicability.
We show that not the idea of LM fine-tuning but our lack of knowledge of how to train them effectively has been hindering wider adoption. LMs overfit to small datasets and suffered catastrophic forgetting when fine-tuned with a classifier. Compared to CV, NLP models are typically more shallow and thus require different fine-tuning methods.
Instead of using the same learning rate for all layers of the model, discriminative fine-tuning allows us to tune each layer with different learning rates.
T 是迭代次数，这里实际上是 $epochs \times \text{number of per epoch}$
cut_frac 是增加学习率的迭代步数比例
cut 是学习率增加和减少的临界迭代步数
p 是一个分段函数，分别递增和递减
ratio 表示学习率最小时，与最大学习率的比例。比如 t=0时，p=0, 那么 $\eta_0=\dfrac{\eta_{max}}{ratio}$
作者通过实验发现，cut_frac=0.1, ratio=32, $\eta_max=0.01$
Target task classifier fine-tuning
针对分类任务，加上 two additional linear blocks.
concat pooling
gradul unfreezing
逐渐 unfreeze layers:
We first unfreeze the last layer and fine-tune all unfrozen layers for one epoch. We then unfreeze the next lower frozen layer and repeat, until we finetune all layers until convergence at the last iteration.
BPTT for Text Classification
backpropagation through time(BPTT)
We divide the document into fixed length batches of size b. At the beginning of each batch, the model is initialized with the final state of the previous batch; we keep track of the hidden states for mean and max-pooling; gradients are back-propagated to the batches whose hidden states contributed to the final prediction. In practice, we use variable length backpropagation sequences (Merity et al., 2017a).
In practice, the players are represented with deep neural nets and updates are made in parameter space, so these results, which depend on convexity, do not apply
Heuristic, non-saturating game
Minimizing the cross-entropy between a target class and a classifier’s predicted distribution is highly effective because the cost never saturates when the classifier has the wrong output.
In the minimax game, the generator minimizes the log-probability of the discriminator being correct. In this game, the generator maximizes the logprobability of the discriminator being mistaken.
in practice, both stochastic gradient descent on the KL divergence and the GAN training procedure will have some variance around the true expected gradient due to the use of sampling (of x for maximum likelihood and z for GANs) to construct the estimated gradient.
所以，从这个视角来看， Maximum likehood 倾向于给 data 出现的位置更高的概率，而 reverse KL 则倾向于给没有出现 data 的位置较低的概率。所以 $D_{KL}(p_{model}||q_{data})$ 可以生成更棒的样本，因为模型不会生成不常见的样本，因为数据之间具有欺骗性的模态。
GANs 通常选择少量的 mode 来生成样本，这个少量指的是小于模型的能力。 而 reverse KL 则是选择更可能多的 mode of the data distribution 在模型能力范围内。它通常不会选择更少的 mode. 这也解释了 mode collapse 并不是散度选择的原因。
Altogether, this suggests that GANs choose to generate a small number of modes due to a defect in the training procedure, rather than due to the divergence they aim to minimize.
所以，GANs 选择少量的 mode 是因为训练过程中的其他缺陷，而不是 散度 选择的问题。
Comparison of cost functions
生成对抗网络可以看做一种 reinforcement learning. 但是$j^{(G)}$ 并没有直接参考 training data，所有关于 training data 的信息都来自于 判别器 的学习。
所以和传统的强化学习是有区别的：
比较不同的 cost function：
$D(G(z))$ 表示 判别器 给 generate sample 为真的概率。
在左侧，Minimax 和 Maximum likehood 都趋向于饱和。也就是说，当一个样本很明显为 fake 时，cost 接近于 0.
Maximum likelihood also suffers from the problem that nearly all of the gradient comes from the right end of the curve, meaning that a very small number of samples dominate the gradient computation for each minibatch. This suggests that variance reduction techniques could be an important research area for improving the performance of GANs, especially GANs based on maximum likelihood.
The DCGAN architecture
DCGAN 的结构。
GAN，NCE， MLE 的对比
相同点：
MiniMax GAN 和 NCE 的 cost function 相同
不同点：
更新策略不一样，GAN 和 MLE 都是梯度下降，而 MLE copies the density model learned inside the discriminator and converts it into a sampler to be used as the generator. NCE never updates the generator; it is just a fixed source of noise.
When the task-specific dataset is with more than two different labels,we should re-train a label size compatible label embeddings layer instead of directly fine-tuning the pre-trained one.
For this method, the self-attention mechanism is modified to explicitly learn the relative positional information between every two sequence positions. As a result, the input sequence is modeled as a labeled, directed, fully-connected graph, where the labels represent positional information. A tunable parameter k is also introduced that limits the maximum distance considered between two sequence positions. [Shaw et al., 2018] hypothesized that this will allow the model to generalize to longer sequences at test time.
对于 GPT 如果它使用双向，那么模型就能准确的学到到句子中的下一个词是什么，并能 100% 的预测出下一个词。比如 “I love to work on NLP.” 在预测 love 的下一个词时，模型能看到 to，所以能很快的通过迭代学习到 “to” 100% 就是 love 的下一个词。所以，这导致模型并不能学到想要的东西（句法、语义信息）。
那么 BERT 是怎么处理双向这个问题的呢？ 它改变了训练语言模型的任务形式。提出了两种方式 “masked language model” and “next sentence generation”. 再介绍这两种训练方式之前，先说明下输入形式。
If the model had been trained on only predicting ‘<MASK>’ tokens and then never saw this token during fine-tuning, it would have thought that there was no need to predict anything and this would have hampered performance. Furthermore, the model would have only learned a contextual representation of the ‘<MASK>’ token and this would have made it learn slowly (since only 15% of the input tokens are masked). By sometimes asking it to predict a word in a position that did not have a ‘<MASK>’ token, the model needed to learn a contextual representation of all the words in the input sentence, just in case it was asked to predict them afterwards.
Well, ideally we want the model’s representation of the masked token to be better than random. By sometimes keeping the sentence intact (while still asking the model to predict the chosen token) the authors biased the model to learn a meaningful representation of the masked tokens.
closed set recognition 天然的特性使得它必须选择一个类别作为预测对象。但是实际场景下， recognition system 必须学会 reject unknown/unseen classes 在 testing 阶段。
于是乎，作者提出了一个新的 model layer, OpenMax, 能够估计一个样本输入是来自于 unknown class 的概率。
A key element of estimating the unknown probability is adapting Meta-Recognition concepts to the activation patterns in the penultimate layer of the network.
OpenMax incorporates likelihood of the recognition system failure. This likelihood is used to estimate the probability for a given input belonging to an unknown class. For this estimation, we adapt the concept of Meta-Recognition[22, 32, 9] to deep networks. We use the scores from the penultimate layer of deep networks (the fully connected layer before SoftMax, e.g., FC8) to estimate if the input is “far” from known training data. We call scores in that layer the activation vector(AV).
关于 OpenMax 如果实现的简单总结，回过头在看。
A key insight in our opening deep networks is noting that “open space risk” should be measured in feature space rather than in pixel space.
一个重要的观点是，在 open deep networks 里面， open space risk 应该是从特征空间 feature space 的角度出发的， 而不是 pixel space. 也就是神经网络判断是不是 unknown, 应该是从 feature 的角度来看的。
We show that an extreme-value meta-recognition inspired distance normalization process on the overall activation patterns of the penultimate network layer provides a rejection probability for OpenMax normalization for unknown images, fooling images and even for many adversarial images.
Open set deep networks
Building on the concepts of open space risk, we seek to choose a layer (feature space) in which we can build a compact abating probability model that can be thresholded to limit open space risk.
基于 open space risk 的概念，提出了 compact abating probability model 能限制 open space risk.
multi-classes meta-recognition
作者先简单介绍了一下前人的工作:
. Prior work on meta-recognition used the final system scores, analyzed their distribution based on Extreme Value Theory (EVT) and found these distributions follow Weibull distribution.
感觉看懂这部分先要理解极值理论(Extreme value theory).
from wikipedia:
It seeks to assess, from a given ordered sample of a given random variable, the probability of events that are more extreme than any previously observed.
它试图从给定随机变量的给定有序样本中评估比先前观察到的任何事件更极端的事件的概率.
然后是 极值分布 的一种 Weibull distribution
所以 Weibull distribution 就是从整个分布中取最极端的例子 sampling top-n score，然后的到的分布。
We take the approach that the network values from penultimate layer (hereafter the Activation Vector (AV)), are not an independent per-class score estimate, but rather they provide a distribution of what classes are “related.”
In real-world recognition/classification tasks, limited by various objective factors, it is usually difficult to collect training samples to exhaust all classes when training a recognizer or classifier. A more realistic scenario is open set recognition (OSR), where incomplete knowledge of the world exists at training time, and unknown classes can be submitted to an algorithm during testing, requiring the classifiers not only to accurately classify the seen classes, but also to effectively deal with the unseen ones.
现实中对于分类任务，不可能在训练集中穷尽所有类别。更实际的情况是 open set recognition (OSR). 在训练阶段包含的是不完整的 knowledge of world. 在测试阶段会出现 unknown 类别。这需要分类器不仅能准确的识别在训练阶段已经见到过的类别，也能有效的处理没有见过的类别, 比如 rejection 或者归类为 unknown.
This paper provides a comprehensive survey of existing open set recognition techniques covering various aspects ranging from related definitions, representations of models, datasets, experiment setup and evaluation metrics. Furthermore, we briefly analyze the relationships between OSR and its related tasks including zero-shot, one-shot (few-shot) recognition/learning techniques, classification with reject option, and so forth. Additionally, we also overview the open world recognition which can be seen as a natural extension of OSR. Importantly, we highlight the limitations of existing approaches and point out some promising subsequent research directions in this field.
这篇综述覆盖了相关的定义、模型、实验以及验证指标。更多地，还分析了 与OSR 相关的任务 zero-shot, one-shot 识别，以及 rejection. 额外地，还概述了 open world recognition 可以看作 OSR 的扩展。更重要的是，作者说明了当前一些方法的限制，并指出了未来研究的一些方向。
Introduction
a more realistic scenario is usually open and non-stationary such as driverless, fault/medical diagnosis, etc., where unseen situations can emerge unexpectedly, which drastically weakens the robustness of these existing methods.
更现实的场景是 开放的和非静态 的。
To meet this challenge, several related research directions actually have been explored including lifelong learning [1], [2], transfer learning [3]–[5], domain adaptation [6], zero-shot [7]–[9], one-shot (few-shot) [10]–[16] recognition/learning and open set recognition/classification [17]–[19], and so forth.
涉及到的领域： lifelong learning, transfer learning, domain adaption, zero-shot, one-shot, open set recogntion.
recognition should consider four basic categories of classes as follows:
known known: train/dev 中有标签的样本，包括正负类别，并且有相关的语义信息。
known unknown: train/dev 中有标签的样本，负类，没有相关的语义信息。
unknown known: test 中没有出现在 train 中的样本，但是有相关的语义信息。比如，train 中有猫，然后 test 中有另外一种猫科动物，那么动物这个样本是有意义的吧？？？
unknown unknown: test 中没有出现在 train 中的样本，并且没有任何相关的语义信息。
Unlike the traditional classification, zero-shot learning (ZSL) can identify unseen classes which have no available observations in training. However, the available semantic/attribute information shared among all classes including seen and unseen ones are needed.
zero-shot 是针对 unknown known, 也就是包含了语义信息。
The ZSL mainly focuses on the recognition of the unknown known classes defined above. Actually, such a setting is rather restrictive and impractical, since we usually know nothing about the testing samples which may come from known known classes or not.
unknown known 这种设定很有限，并且不切实际。因为我们很难知道 test 中的样本是否是包含了语义信息，无法判断是 unknown known or unknown unknown.
comparision between open set recognition and traditional classification
Via these decision boundaries, samples from an unknown unknown class are labeled as ”unknown” or rejected rather than misclassified as known known classes.
Basic notation and related definition
经验风险函数：
$L(x, y, f(x)) \ge 0$ 是 loss function. P(x,y) 是对应样本 (x, y) 的概率，通常这个联合分布的概率我们是不知道的，因为我们无法确定自然界中样本空间(label space)到底是个什么分布。
[李航，机器学习] 中关于风险函数的定义：
损失函数度量一次模型预测的好坏，风险函数度量平均意义下模型预测的好坏。
以前看不懂这一部分，现在只想说： Perfect!
Therefore, traditional recognition/classification approaches minimize the empirical risk instead of the ideal risk RI by using other knowledge, such as assuming that the label space is at least locally smooth and regularizing the empirical risk minimization.
传统的分类方法是根据其他的外部知识来最小化经验风险，比如 label space 是光滑的，然后使用正则化最小经验风险（也就是上面所说的结构风险函数）。
Note that traditional recognition problem is usually performed under the closed set assumption. When the assumption switches to open environment/set scenario with the open space, other things should be added since intuitively there is some risk in labeling sample in the open space as any known known classes. This gives such an insight for OSR that we do know something else: we do know where known known classes exist, and we know that in open space we do not have a good basis for assigning labels for the unknown unknown classes.
传统的识别是假设在固定的样本空间下(known known). 当转换到开放式场景下，我们很敏感的意识到需要给 label space 加点 risk。。我们知道 known known classes 是存在的，我们也知道我们并没有这样一个 basis 去给 unknown unknown 打标签。
这篇 paper 是把 class of interest 当作一个类，然后所有的 unknown/known 当作很多 classes, 也就是 1-vs-set.
To improve the overall open set recognition error, our 1-vs-set formulation balances the unknown classes by obtaining a core margin around the decision boundary A from the base SVM, specializing the resulting half-space by adding another plane $\Omega$ and then generalizing or specializing the two planes (shown in Fig. 2) to optimize empirical and open space risk. This process uses the open set training data and the risk model to define a new “open set margin.” The second plane $\Omega$ allows the 1-vs-set machine to avoid the overgeneralization that would misclassify the raccoon in Fig. 2. The overall optimization can also adjust the original margin with respect to A to reduce open space risk, which can avoid negatives such as the owl.
使用了两个超平面，去分隔 Negatives/positivecs/unknown.
While we do not know the joint distribution $P(x, y)$ in, one way to look at the open space risk is as a weak assumption: Far from the known data the Principle of Indifference [8] suggests that if there is no known reason to assign a probability, alternatives should be given equal probability. In our case, this means that at all points in open space, all labels (both known and unknown) are equally
likely, and risk should be computed accordingly. However, we cannot have constant value probabilities over infinite spaces—the distribution must be integrable and integrate to 1. We must formalize open space differently (e.g., by ensuring the problem is well posed and then assuming the probability is proportional to relative Lebesgue measure [9]). Thus, we can consider the measure of the open space to the full space, and define our risk penalty proportional to such a ratio.
无法知道联合分布 $P(x, y)$, 作者假设所有的样本概率是相等的，但是向量空间中样本总数是不确定的，所以作者定义一个比例来描述 在 open space 中出现 unknown 的危险惩罚系数。
where open space risk is considered to be the fraction (in terms of Lebesgue measure) of positively labeled open space compared to the overall measure of positively labeled space (which includes the space near the positive examples).
open space risk 是开放空间中 positive label 的总数与总体空间中 positive label 的总体度量。
不太懂。。问题还是不知道怎么度量？ unknown 的类别能确定？？？
openness
openness，用来表征数据集的开放程度：
$C_{TR}$ 是训练集中的类别数，越大，开放程度越小。
$C_{TE}$ 测试集中的类别数。
The Open Set Recognition Problem
our goal is to balance the risk of the unknown in open space with the empirical (known) risk. In this sense, we formally define the open set recognition problem as follows:
我们的目的是平衡 the risk of unknown 出现在基于 known classes 计算的到的 open space 的 empirical risk。
怎么理解呢？就是传统的风险函数都是只考虑了经验风险，也就是完全基于训练数据的。但是在 open space 里面，我们还要测试时会出现的 unknown，所以在 风险函数的设置的同时，就要考虑到 unknown 的存在。也就是前面的 open space risk.
a categorization of OSR techniques
问题的关键在于如何将 公式（4）open space risk 合并到模型中去。然后大佬们提出各式各样的模型，主要分为 discriminative model and generative models.
更进一步，可以分为：five categories (Table II):
Traditional ML-based
Deep Network-based
Adversarial Learning-based
EVT-based
Dirichlet Process-based OSR models
Deep Neural Network-based OSR Models
大佬们的杰作，感觉都挺新的，新坑？
提出了 OpenMax,使用 deep networks, 还是用 softmax 损失函数来最小化 交叉熵 cross entropy loss. 然后在网络的倒数第二层（softmax 的前一层？）得到每一个正分类的 mean activate vector(MAV).
然后是根据 Weibull districution 去 redistribution 以及重新分类等等接下来的操作还是看相应 的 paper 吧。
the OpenMax effectively addressed the challenge of the recognition for fooling/rubbish and unrelated open set images. However, as discussed in [71], the OpenMax fails to recognize the adversarial images which are visually indistinguishable from training samples but are designed to make deep networks produce high confidence but incorrect answers [96], [98].
OpenMax 有效的解决了 不相关的 open set images 的问题，但是却无法有效区分对抗生成样本。
Actually, the authors in [72] have indicated that the OpenMax is susceptible to the adversarial generation techniques directly working on deep representations. Therefore, the adversarial samples are still a serious challenge for open set recognition. Furthermore, using the distance from MAV, the cross entropy loss function in OpenMax does not directly incentivize projecting class samples around the MAV. In addition to that, the distance function used in testing is not used in training, possibly resulting in inaccurate measurement in that space [73]. To address this limitation, Hassen and Chan [73] learned a neural network based representation for open set recognition, which is similar in spirit to the Fisher Discriminant, where samples from the same class are closed to each other while the ones from different classes are further apart, leading to larger space among known known classes for unknown unknown classes’ samples to occupy.
交叉熵并不能有效的将类别映射到相应的 MAV 中，因为在测试集中的 distence function 跟在 training set 里面是不一样的，这会导致不准确的判别。基于此，[73]提出了 Fisher 判别，从同一个类别中采样，使得unknown unknown 和 known known 的间距很大。
OpenMax to text classification
Deep Open classifier
tWiSARD
hidden unknown unknown classes
Adversarial Learning-based OSR Models
Note that the main challenge for open set recognition is the incomplete class knowledge existing in training, leading to the open space risk when classifiers encounter unknown unknown classes during testing. Fortunately, the adversarial learning technique can account for open space to some extent by adversarially generating the unknown unknown class data according to the known known class knowledge, which undoubtedly provides another way to tackle the challenging multiclass OSR problem.
open set recognition 最大的挑战是 training 中不完整的 knowledge， 在 testing 中遇到 unknown unknown 导致 open space risk.
而对抗训练网络在某种程度上根据 known known 生成 unknown unknown，提供了另外一种方式解决 OSR 问题。
EVT-based OSR Models
As a powerful tool to increase the classification performance, the statistical Extreme Value Theory (EVT) has recently achieved great success due to the fact that EVT can effectively model the tails of the distribution of distances between training observations using the asymptotic theory[100].
不是很懂这个理论，给出几篇 paper 吧
Remark: As mentioned above, almost all existing OSR methods adopt the threshold-based classification scheme, where recognizers in decision either reject or categorize the input samples to some known known class using empirically set threshold. Thus the threshold plays a key role. However, the selection for it usually depends on the knowledge of known known classes, inevitably incurring risks due to lacking available information from unknown unknown classes [57]. This indicates the threshold-based OSR methods still face serious challenges.
基于 known known 得到的 threshold 因为缺乏 unknown unknown 的信息，不可避免的会造成 risk, 这也是基于 threshold 这类方法所面临的困难。
Dirichlet Process-based OSR Models (生成模型)
Dirichlet process (DP) [104]–[108] considered as a distribution over distributions is a stochastic process, which has been widely applied in clustering and density estimation problems as a nonparametric prior defined over the number of mixture components. Furthermore, this model does not overly depend on training samples and can achieve adaptive change as the data changes, making it naturally adapt to the open set recognition scenario. In fact, researchers have begun the related research
Dirichlet 过程作为一种基于混合模型的非参数方法广泛用于聚类，参数估计。这种模型不需要依赖于 training，可以随着 dataset 的变化而自适应的变化，这使得它能有效的适用于 open set 的场景。
对生成模型不是很熟。。
Remark: Instead of addressing the OSR problem from the discriminative model perspective, CD-OSR actually reconsiders this problem from the generative model perspective due to the use of HDP, which provides another research direction for open set recognition. Furthermore, the collective decision strategy for OSR is also worth further exploring since it not only takes the correlations among the testing samples into account but also provides a possibility for new class discovery, whereas single-sample decision strategy2 adopted by other existing OSR methods can not do such a work since it can not directly tell whether the single rejected sample is an outlier or from new class.
Beyond open set Recognition
关于 open set recognition 如果仅仅考虑静态的 set，意义不是很大。以及，只对 unknown unknown 进行 rejection 也是不够的。为此，有人提出 open world recognition.
open world recognition (OWR), where a recognition system should perform four tasks:
detecting unknown unknown classes
choosing which samples to label for addition to the model
labelling those samples
updating the classifier
Remark: As a natural extension of OSR, the OWR faces more serious challenges which require it not only to have the ability to handle the OSR task, but also to have minimal
downtime, even to continuously learn, which seems to have the flavor of lifelong learning to some extent. Besides, although some progress regarding the OWR has been made, there is still a long way to go.
Experiment Setup: In open set recognition, most existing experiments are carried out on a variety of recastes multi-class benchmark datasets. Specifically, taking the Usps dataset as an example, when it is used for OSR problem, one can randomly choose S distinct labels as the known known classes, and vary openness by adding a subset of the remaining labels.
Instead, the computations of Precision and Recall in it are only for available known known classes. Additionally, the work [67] has indicated that although the computations of Precision and Recall are only for available known known classes, the FN and FP also consider the false unknown unknown classes and false known known classes by taking into account the false negative and the false positive, and we refer the reader to [67] for more details.
contains the macro-Precision and micro-Precision while Recall includes the macro-Recall and micro-Recall, which leads to the corresponding macro-F-measure and micro-F-measure. Nevertheless, whether it is macro-F-measure or micro-F-measure, the higher their values, the better the performance of the corresponding OSR model.
Youden’s index for OSR
$$J= \text{Recall}+S-1$$
其中 S 是真负类率： $S=\dfrac{TN}{TN+FP}$
future research directions
About modeling
大部分工作都是基于判别模型来做的，只有少部分是基于生成模型，也许生成模型会更有探索空间。
OSR 的主要挑战是传统的分类器是在 closed-set 场景下获得的，一旦 unknown unknown class 落入这个空间，将永远无法被正确的分类。
modeling known known classes
如果得到的 known known class 没有被过拟合，那么这样的分类器就能有效的区分出 unknown unknown. 所以聚类和分类算法的结合会是不错的方向。关于 clustering 和 classification 的 unified learning framework:
这两篇 paper 依旧是在 closed-set 下做的，所以需要你去尝试。。。
modeling unknown unknown classes
似乎在只有 known known classes 的情况下是很难去学习 unknown unknown 的类的性质的。但是可以通过对抗学习来生成 unknown unknown 也是不错的方向。
顺便作者还提了下 transductive leanring，以及基于 Dirichlet process 的自适应行，CD-OSR、Dirichlet processed-based OSR 也是值得探索的。
About rejecting
大部分的工作都是 reject unknown unknown classes，而没有后续的工作了。只有少量的 [66][67]进行了后续的工作，比如 new classes discovery.
About the decision
所有的 OSR 模型都是用来识别单个样本的，但是一个决策的决定并没有考虑样本之间的相关性。所以 collective decision 不仅在 testing 时考虑相关性，同时还能发现 new classes.
Open set + ‘sth’
As open set scenario is a more practical assumption for the real-world classification/recognition tasks, it can naturally be combined with various fields involving classification/recognition such as semi-supervised learning, domain adaptation, active learning, multi-task learning, multi-label learning, multi-view learning, and so forth. For example, [124]–[126] recently introduced this scenario into domain adaptation, while [127] explored the open set classification in active learning field. Therefore, many interesting works are worth looking forward to.
看起来是个不错的方向。。
Generalized Open Set Recognition
利用 side-information,比如 unknown unknwon 和 known known 会有共同的语义信息(semantic/attribute information).
Appending semantic/attribute information
In fact, a lot
of semantic/attibute information is shared between the known known and the unknown unknown classes. Therefore, we can fully utilize this kind of information to ’cognize’ the unknown unknown classes, or at least to provide a rough semantic/attribute description for the corresponding unknown unknown classes instead of simply rejecting them.
利用语义信息去意识到 unknown unknwon，而不是简单的 reject.
但是要注意区分 open set recognition 和 ZSL(zero-shot learning) 的区别：
The $\text{side-information}^1$ in ZSL denotes the semantic/attribute information shared among all classes including known known and unknown known classes.
where the $\text{side-information}^4$ denotes the available semantic/attribute information only for known known classes
感觉这个 side-information 的界限很难确定啊？Generalized Open Set Recognition 的这个范围似乎很难实现， 怎么可能出现在 training 中的 semantice information 完全不出现在 unknown unknown 中呢。。
还有一些相似的工作：
Using other available side-information
**The main reason for open space risk is that the traditional classifiers trained under closed set scenario usually divide over-occupied space for known known classes, thus inevitably resulting in misclassifications once the unknown unknown class samples
fall into the space divided for some known known class.** From this perspective, the open space risk will be reduced as the space divided for those known known classes decreases by
using other side-information like universum [135], [136] to shrink their regions as much as possible.
虽然感觉很扯淡。。但是还是有人做啊，不过关于 open space risk 的定义可以在看一遍。。
Relative Open Set Recognition
感觉这个还挺有意思的。疾病的诊断，所有的样本空间都可以区分为 sick or no sick, 所以仅仅是判断有没有病，那么这是个 closed set 问题。但是如果我们要进一步判断疾病的类型，那么有可能出现 unseen disease in training.
Knowledge Integration for Open Set Recognition
In fact, the incomplete knowledge of the world is universal, especially for the single individuals: something you know does not mean I also know.
how to integrate the classifiers trained on each sub-knowledge set to further reduce the open space risk will be an interesting yet challenging topic in the future work, especially for such a situation: we can only obtain the classifiers trained on corresponding sub-knowledge sets, yet these sub-knowledge sets are not available due to the privacy protection of data.
We introduce CoQA, a novel dataset for building Conversational Question Answering systems.1 Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains.
CoQA, 对话式阅读理解数据集。从 7 个不同领域的 8k 对话中获取的 127k 问答对。
The questions are conversational, and the answers are free-form text with their corresponding evidence highlighted in the passage.
We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning.
CoQA 跟传统的 RC 数据集所面临的挑战不一样，主要是指代和推理。
We ask other people a question to either seek or test their knowledge about a subject. Depending on their answer, we follow up with another question and their answer builds on what has already been discussed. This incremental aspect makes human conversations succinct. An inability to build up and maintain common ground in this way is part of why virtual assistants usually don’t seem like competent conversational partners.
In CoQA, a machine has to understand a text passage and answer a series of questions that appear in a conversation. We develop CoQA with three main goals in mind.
The first concerns the nature of questions in a human conversation. Posing short questions is an effective human conversation strategy, but such questions are a pain in the neck for machines.
第一点：人类在对话时，会提出很简短的问题，但这对于机器来说却很难。比如 Q5 “Who?”
The second goal of CoQA is to ensure the naturalness of answers in a conversation. Many existing QA datasets restrict answers to a contiguous span in a given passage, also known as extractive answers (Table 1). Such answers are not always natural, for example, there is no extractive answer for Q4 (How many?) in Figure 1. In CoQA, we propose that the answers can be free-form text (abstractive answers), while the extractive spans act as rationales for the actual answers. Therefore, the answer for Q4 is simply Three while its rationale is spanned across multiple sentences.
The third goal of CoQA is to enable building QA systems that perform robustly across domains. The current QA datasets mainly focus on a single domain which makes it hard to test the generalization ability of existing models.
第三点：数据来自多种 domain，提高泛化性。
Dataset collection
数据集具体详情：
It consists of 127k conversation turns collected from 8k conversations over text passages (approximately one conversation per
passage). The average conversation length is 15 turns, and each turn consists of a question and an answer.
It contains free-form answers. Each answer has an extractive rationale highlighted in the passage.
Its text passages are collected from seven diverse domains — five are used for in-domain evaluation and two are used for out-of-domain
evaluation.
Almost half of CoQA questions refer back to conversational history using coreferences, and a large portion requires pragmatic reasoning making it challenging for models that rely on lexical cues alone.
The best-performing system, a reading comprehension model that predicts extractive rationales which are further fed into a sequence-to-sequence model that generates final answers, achieves a F1 score of 65.1%. In contrast, humans achieve 88.8% F1, a superiority of 23.7% F1, indicating that there is a lot of headroom for improvement.
Baseline 是将抽取式阅读理解模型转换成 seq2seq 形式，然后从 rationale 中获取答案，最终得到了 65.1% 的 F1 值。
question and answer collection
We want questioners to avoid using exact words in the passage in order to increase lexical diversity. When they type a word that is already present in the passage, we alert them to paraphrase the question if possible.
For the answers, we want answerers to stick to the vocabulary in the passage in order to limit the number of possible answers. We encourage this by automatically copying the highlighted text into the answer box and allowing them to edit copied text in order to generate a natural answer. We found 78% of the answers have at least one edit such as changing a word’s case or adding a punctuation.
Not all passages in these domains are equally good for generating interesting conversations. A passage with just one entity often result in questions that entirely focus on that entity. Therefore, we select passages with multiple entities, events and pronominal references using Stanford CoreNLP (Manning et al., 2014). We truncate long articles to the first few paragraphs that result in around 200 words.
Table 2 shows the distribution of domains. We reserve the Science and Reddit domains for out-ofdomain evaluation. For each in-domain dataset, we split the data such that there are 100 passages in the development set, 100 passages in the test set, and the rest in the training set. For each out-of-domain dataset, we just have 100 passages in the test set.
Some questions in CoQA may have multiple valid answers. For example, another answer for Q4 in Figure 2 is A Republican candidate. In order to
account for answer variations, we collect three additional answers for all questions in the development and test data.
一个问题可能出现多种回答，因此在dev dataset 和 test dataset 中有三个候选答案。
In the previous example, if the original answer was A Republican Candidate, then the following question Which party does he
belong to? would not have occurred in the first place. When we show questions from an existing conversation to new answerers, it is likely they will deviate from the original answers which makes the conversation incoherent. It is thus important to bring them to a common ground with the original answer.
比如上图中 Q4, 如果回答是 A Republican candidate. 但是整个对话是相关的，所以接下来的问题就会使整个对话显得混乱了。
We achieve this by turning the answer collection task into a game of predicting original answers. First, we show a question to a new answerer, and when she answers it, we show the original answer and ask her to verify if her answer matches the original. For the next question, we ask her to guess the original answer and verify again. We repeat this process until the conversation is complete. In our pilot experiment, the human F1 score is increased by 5.4% when we use this verification setup.
因为机器在学习的时候是有 original answer 进行对比的，同样的这个过程在人工阶段也是需要的，可以减少上诉的混乱情况，answerer 在给出一个答案后，作者会告诉他们是否与 original 匹配，然后直到整个过程完成。
Dataset Analysis
What makes the CoQA dataset conversational compared to existing reading comprehension datasets like SQuAD? How does the conversation flow from one turn to the other? What linguistic phenomena do the questions in CoQA exhibit? We answer these questions below.
在 question 中：
指代词(he, him, she, it, they)出现的更为频繁， SQuAD 则几乎没有。
SQuAD 中 what 几乎占了一半，CoQA 中问题类型则更为多样， 比如 did, was, is, does 的频率很高。
Deep neural networks often work well when they are over-parameterized and trained with a massive amount of noise and regularization, such as weight decay and dropout. Although dropout is widely used as a regularization technique for fully connected layers, it is often less effective for convolutional layers. This lack of success of dropout for convolutional layers is perhaps due to the fact that
activation units in convolutional layers are spatially correlated so information can still flow through convolutional networks despite dropout.
通常深度神经网络在过参数化，并在训练时加上大量的噪声和正则化，比如权重衰减和 dropout，这个时候神经网络能很好的 work. 但是 dropout 对于全链接网络是一个非常有效的正则化技术，它对于卷积神经网络却没啥效果。这可能是因为卷积神经网络的激活是空间相关的，即使 drop 掉部分 unit，信息仍然会传递到下一层网络中去。
Thus a structured form of dropout is needed to regularize convolutional networks. In this paper, we introduce DropBlock, a form of structured dropout, where units in a contiguous region of a feature map are dropped together. We found that applying DropBlock in skip connections in addition to the convolution layers increases the accuracy. Also, gradually increasing number of dropped units during training leads to better accuracy and more robust to hyperparameter choices.
作者为卷积神经网络提出了专门的正则化方式， dropblock. 同时 drop 掉一个连续的空间。作者发现将 dropblock 应用到 ResNet 能有效的提高准确率。同时增加 drop 的概率能提高参数的鲁棒性。
In this paper, we introduce DropBlock, a structured form of dropout, that is particularly effective to regularize convolutional networks. In DropBlock, features in a block, i.e., a contiguous region of a feature map, are dropped together. As DropBlock discards features in a correlated area, the networks must look elsewhere for evidence to fit the data (see Figure 1).
具体的算法很简单，主要关注两个参数的设置： block_size 和 $\gamma$.
block_size is the size of the block to be dropped
$\gamma$ controls how many activation units to drop.
We experimented with a shared DropBlock mask across different feature channels or each feature channel has its DropBlock mask. Algorithm 1 corresponds to the latter, which tends to work better in our experiments.
Similar to dropout we do not apply DropBlock during inference. This is interpreted as evaluating an averaged prediction across the exponentially-sized ensemble of sub-networks. These sub-networks include a special subset of sub-networks covered by dropout where each network does not see contiguous parts of feature maps.
关于 infer 时， dropblock 的处理和 dropout 类似。
block_size:
In our implementation, we set a constant block_size for all feature maps, regardless the resolution of feature map. DropBlock resembles dropout [1] when block_size = 1 and resembles SpatialDropout [20] when block_size covers the full feature map.
In practice, we do not explicitly set $\gamma$. As stated earlier, $\gamma$ controls the number of features to drop. Suppose that we want to keep every activation unit with the probability of keep_prob, in dropout [1] the binary mask will be sampled with the Bernoulli distribution with mean 1 − keep_prob. However, to account for the fact that every zero entry in the mask will be expanded by block_size2 and the blocks will be fully contained in feature map, we need to adjust $\gamma$ accordingly when we sample the initial binary mask. In our implementation, $\gamma$ can be computed as
The main nuance of DropBlock is that there will be some overlapped in the dropped blocks, so the above equation is only an approximation.
最主要的问题是，会出现 block_size 的重叠。所以上诉公式也只是个近似。
Scheduled DropBlock:
We found that DropBlock with a fixed keep_prob during training does not work well. Applying small value of keep_prob hurts learning at the beginning. Instead, gradually decreasing keep_prob over time from 1 to the target value is more robust and adds improvement for the most values of keep_prob.
In the following experiments, we study where to apply DropBlock in residual networks. We experimented with applying DropBlock only after convolution layers or applying DropBlock after both convolution layers and skip connections. To study the performance of DropBlock applying to different feature groups, we experimented with applying DropBlock to Group 4 or to both Groups 3 and 4.
params_grad = evaluate_gradient ( loss_function , example , params )
params = params - learning_rate * params_grad
Batch gradient descent performs redundant computations for large datasets, as it recomputes gradients for similar examples before each parameter update. SGD does away with this redundancy by performing one update at a time. It is therefore usually much faster and can also be used to learn
online. SGD performs frequent updates with a high variance that cause the objective function to fluctuate heavily.
批梯度下降的计算过于冗余，它在每一次参数更新之前的计算过程中会计算很多相似的样本。随机梯度下降则是每一次参数更新计算一个样本，因此更新速度会很快，并且可以在线学习。但是用于更新的梯度的方差会很大，导致 loss 曲线波动很大。
While batch gradient descent converges to the minimum of the basin the parameters are placed in, SGD’s fluctuation, on the one hand, enables it to jump to new and potentially better local minima. On the other hand, this ultimately
complicates convergence to the exact minimum, as SGD will keep overshooting. However, it has been shown that when we slowly decrease the learning rate, SGD shows the same convergence behaviour as batch gradient descent, almost
certainly converging to a local or the global minimum for non-convex and convex optimization respectively.
reduces the variance of the parameter updates, which can lead to more stable convergence;
减小参数更新的方差，使得收敛更稳定。
can make use of highly optimized matrix optimizations common to state-of-the-art deep learning libraries that make computing the gradient mini-batch very efficient.
能非常好的利用矩阵优化的方式来加速计算，这在各种深度学习框架里面都很常见。
Challenges
Choosing a proper learning rate.
选择合适的学习率。
Learning rate schedules. try to adjust the learning rate during training by e.g. annealing, i.e. reducing the learning rate according to a pre-defined schedule or when the change in objective between epochs falls below a threshold. These schedules and thresholds, however, have to be defined in advance and are thus unable to adapt to a dataset’s characteristics.
the same learning rate applies to all parameter updates. If our data is sparse and our features have very different frequencies, we might not want to update all of them to the same extent, but perform a larger update for rarely occurring
Another key challenge of minimizing highly non-convex error functions common for neural networks is avoiding getting trapped in their numerous suboptimal local minima. Dauphin et al. [5] argue that the difficulty arises in fact not from local minima but from saddle points, i.e. points where one dimension slopes up and another slopes down. These saddle points are usually surrounded by a plateau of the same error, which makes it notoriously hard for SGD to escape, as the gradient is close to zero in all dimensions.
对于非凸损失函数的优化问题，需要避免陷入其众多的次优局部极小值。Dauphin et al. [5] 则认为， 相比局部极小值，鞍点的是更难解决的问题。鞍点是一个维度上升，一个维度下降。详细的关于鞍点以及 SGD 如何逃离鞍点可参考：知乎：如何逃离鞍点 .
Gradient descent optimization algorithms
Momentum [17] is a method that helps accelerate SGD in the relevant direction and dampens oscillations as can be seen in Figure 2b. It does this by padding a fraction $gamma$ of the update vector of the past time step to the current
update vector.
Momentum
paper: [Neural networks :
the official journal of the International Neural Network Society]()
The momentum term increases for dimensions whose gradients point in the same directions and reduces updates for dimensions whose gradients change directions.
paper: [Yurii Nesterov. A method for unconstrained convex minimization problem
with the rate of convergence o(1/k2).]()
We would like to have a smarter ball, a ball that has a notion of where it is going so that it knows to slow down before the hill slopes up again. Nesterov accelerated gradient (NAG) [14] is a way to give our momentum term this kind of prescience.
paper: [Adaptive Subgradient Methods for Online Learning
and Stochastic Optimization]()
Adagrad [8] is an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing larger updates for infrequent and smaller updates for frequent parameters. For this reason, it is well-suited for dealing with sparse data.
对于不同的参数，自适应的调整对应的梯度大小。对低频参数或特征，使其更新的梯度较大，对高频的参数或特征，使其更新的梯度较小。比如在训练 Glove 词向量时，低频词在某一步迭代中可能并没有参与 loss 的计算，所以更新的会相对较慢，所以需要人为的增大它的梯度。
In addition to storing an exponentially decaying average of past squared gradients $v_t$ like Adadelta and RMSprop, Adam also keeps an exponentially decaying average of past gradients $m_t$, similar to momentum:
similar like momentum:
$$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$$
similar like autograd/RMSprop:
$$v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$$
$m_t$ and $v_t$ are estimates of the first moment (the mean) and the second moment (the uncentered variance) of the gradients respectively, hence the name of the method. As $m_t$ and $v_t$ are initialized as vectors of 0’s, the authors of Adam observe that they are biased towards zero, especially during the initial time steps, and especially when the decay rates are small (i.e. β1 and β2 are close to 1). They counteract these biases by computing bias-corrected first and second moment estimates:
$$\hat m_t=\dfrac{m_t}{1-\beta^t_1}$$
$$\hat v_t=\dfrac{v_t}{1-\beta^t_2}$$
They then use these to update the parameters just as we have seen in Adadelta and RMSprop, which
Norms for large p values generally become numerically unstable, which is why $l_1$ and $l_2$ norms are most common in practice. However, $l_{\infty}$ also generally exhibits stable behavior. For this reason, the authors propose AdaMax [10] and show that $v_t$ with $l_{\infty}$ converges to the following more stable value. To avoid confusion with Adam, we use ut to denote the infinity norm-constrained $v_t$:
Note that as $\mu_t$ relies on the max operation, it is not as suggestible to bias towards zero as $m_t$ and $v_t$ in Adam, which is why we do not need to compute a bias correction for ut. Good default values are again:
$$\eta = 0.002, \beta_1 = 0.9, \beta_2 = 0.999.$$
Visualization of algorithms
we see the path they took on the contours of a loss surface (the Beale function). All started at the same point and took different paths to reach the minimum. Note that Adagrad, Adadelta, and RMSprop headed off immediately in the right direction and converged similarly fast, while Momentum and NAG were led off-track, evoking the image of a ball rolling down the hill. NAG, however, was able to correct its course sooner due to its increased responsiveness by looking ahead and headed to the minimum.
如果目标函数是 Beale 这种类型的函数，自适应优化算法能更直接的收敛到最小值。而 Momentum 和 NAG 则偏离了轨道，就像球从山上滚下一样，刹不住车。但是 NAG 因为对未来具有一定的预见性，所以能更早的纠正从而提高其响应能力。
shows the behaviour of the algorithms at a saddle point, i.e. a point where one dimension has a positive slope, while the other dimension has a negative slope, which pose a difficulty for SGD as we mentioned before. Notice here that SGD, Momentum, and NAG find it difficulty to break symmetry, although the latter two eventually manage to escape the saddle point, while Adagrad, RMSprop, and Adadelta quickly head down the negative slope, with Adadelta leading the charge.
各种优化算法鞍点的表现。 Momentum, SGD, NAG 很难打破平衡，而自适应性的算法 Adadelta, RMSprop, Adadelta 能很快的逃离鞍点。
operations, each time casting a scalar feature upon the inner word embeddings. The key idea is to provide a real-valued hint (feature) to a subsequent encoder layer and is targeted at improving the representation learning process.
The key idea of attention is to extract only the most relevant information that is useful for prediction. In the context of textual data, attention learns to weight words and sub-phrases within documents based on how important they are. In the same vein, co-attention mechanisms [5, 28, 50, 54] are a form of attention mechanisms that learn joint pairwise attentions, with respect to both document and query.
Attention is traditionally used and commonly imagined as a feature extractor. It’s behavior can be thought of as a dynamic form of pooling as it learns to select and compose different words to form the final document representation.
This paper re-imagines attention as a form of feature augmentation method. Attention is casted with the purpose of not compositional learning or pooling but to provide hints for subsequent layers. To the best of our knowledge, this is a new way to exploit attention in neural ranking models.
这篇 paper 将 attention 重新设想为一种特征增强的方式，Attention的目的不是组合学习或汇集，而是为后续层提供提示（特征）。这是一种在神经排序模型中的新方法。
An obvious drawback which applies to many existing models is that they are generally restricted to one attention variant. In the case where one or more attention calls are used (e.g., co-attention and intra-attention, etc.), concatenation is generally used to fuse representations [20, 28]. Unfortunately, this incurs cost in subsequent layers by doubling the representation size per call.
很多 paper 中只受限于一种 attention，这明显是不够好的。也有使用多种 attention 的，比如 co-attention 和 intra-attention， 然后直接拼接起来。这样会使得接下来的 modeling layer 维度加倍。。（在 ai challenge 的比赛中就是这么干的。。没啥不好啊。。）
The rationale for desiring more than one attention call is intuitive. In [20, 28], Co-Attention and Intra-Attention are both used because each provides a different view of the document pair, learning high quality representations that could be used for prediction. Hence, this can significantly improve performance.
Moreover, Co-Attention also comes in different flavors and can either be used with extractive max-mean pooling [5, 54] or alignment-based pooling [3, 20, 28]. Each co-attention type produces different document representations. In max-pooling, signals are extracted based on a word’s largest contribution to the other text sequence. Mean-pooling calculates its contribution to the overall sentence. Alignment-pooling is another flavor of co-attention, which aligns semantically similar sub-phrases together.
Our approach is targeted at serving two important purposes: (1) It removes the need for architectural engineering of this component by enabling attention to be called for an arbitrary k times with hardly any consequence and (2) concurrently it improves performance by modeling multiple views via multiple attention calls. As such, our method is in similar spirit to multi-headed attention, albeit efficient. To this end, we introduce Multi-Cast Attention Networks (MCAN), a new deep learning architecture for a potpourri of tasks in the question answering and conversation modeling domains.
In our approach, attention is casted, in contrast to the most other works that use it as a pooling operation. We cast co-attention multiple times, each time returning a compressed scalar feature that is re-attached to the original word representations. The key intuition is that compression enables scalable casting of multiple attention calls, aiming to provide subsequent layers with a hint of not only global knowledge but also cross sentence knowledge.
Figure 1: Illustration of our proposed Multi-Cast Attention Networks (Best viewed in color). MCAN is a wide multi-headed attention architecture that utilizes compression functions and attention as features.
highway encoders can be interpreted as data-driven word filters. As such, we can imagine them to parametrically learn which words have an inclination to be important and not important to the task at hand. For example, filtering stop words and words that usually do not contribute much to the prediction. Similar to recurrent models that are gated in nature, this highway encoder layer controls how much information (of each word) is flowed to the subsequent layers.
Co-Attention [50] is a pairwise attention mechanism that enables
attending to text sequence pairs jointly. In this section, we introduce four variants of attention, i.e., (1) max-pooling, (2) mean-pooling, (3) alignment-pooling, and finally (4) intra-attention (or self attention).
each pooling operator has different impacts and can be intuitively understood as follows: max-pooling selects each word based on its maximum importance of all words in the other text. Mean-pooling is a more wholesome comparison, paying attention to a word based on its overall influence on the other text. This is usually dataset-dependent, regarded as a hyperparameter and is tuned to see which performs best on the held out set.
Intuitively, what is achieved here is that we are modeling the influence of co-attention by comparing representations before and after co-attention. For soft-attention alignment, a critical note here is that x and $\overline x$ (though of equal lengths) have ‘exchanged’ semantics. In other words, in the case of q, $\overline q$ actually contains the aligned representation of d.
Compression Function
The rationale for compression is simple and intuitive - we do not want to bloat subsequent layers with a high dimensional vector which consequently incurs parameter costs in subsequent layers. We investigate the usage of three compression functions, which are capable of reducing a n dimensional vector to a scalar.
我们的架构背后的关键思想是促进k个注意力投射，每个投射都用一个实值注意力提示来增强原始词向量。 对于每个query-document对，应用Co-Attention with mean-pooling，Co-Attention with max-Pooling和Co-Attention with alignment-pooling。 此外，将Intra-Attention分别单独应用于query和document。 每个注意力投射产生三个标量（每个单词），它们与词向量连接在一起。最终的投射特征向量是 $z\in R^{12}$。
As such, the key idea behind casting attention as features right before this layer is that it provides the LSTM encoder with hints that provide information such as (1) longterm and global sentence knowledge and (2) knowledge between sentence pairs (document and query).
对文本来说，看到 CNN 我们首先想到的是 cnn 能有效利用局部信息，提取出局部特征，所以适合做文本分类。但是对于 机器翻译、阅读理解这样的需要考虑全局信息的任务，CNN 似乎看起来并不那么有效。而且在 decoder 的时候，词的生成是 one by one 的，下一个词的生成是依赖于上一个词的。所以在 decoder 中使用 RNN 也是很自然而然的。
Facebook 的这篇 paper 就改变了这些传统的思维，不仅用 CNN 编码全局信息，而且还能 decoder.
Motivation
Multi-layer convolutional neural networks create hierarchical representations over the input sequence in which nearby input elements interact at lower layers while distant elements interact at higher layers.
Hierarchical structure provides a shorter path to capture long-range dependencies compared to the chain structure modeled by recurrent networks, e.g. we can obtain a feature representation capturing relationships within a window of n words by applying only O(n/k) convolutional operations for kernels of width k, compared to a linear number O(n) for recurrent neural networks.
层级结构提供了一个更短的路径来获取长期依赖。比如相距为 n 的两个词，在 rnn 中交互需要的步数是 O(n),在层级 CNN 中需要 O(n/k).这样减少了非线性的操作，降低了梯度消失的情况。所以这两个词的交互效果会更好～
Inputs to a convolutional network are fed through a constant number of kernels and non-linearities, whereas recurrent networks apply up to n operations and non-linearities to the first word and only a single set of operations to the last word. Fixing the number of nonlinearities applied to the inputs also eases learning.
在这篇 paper 中，作者用无监督的方式，来训练语言模型，将 CNN 得到的语言模型与 LSTM 进行对比。
也就是:
$$h_l=(XW+b)\otimes \sigma(XV+c)$$
The output of each layer is a linear projection X ∗ W + b modulated by the gates σ(X ∗ V + c). Similar to LSTMs, these gates multiply each element of the matrix X ∗W+b
and control the information passed on in the hierarchy.
如果是 LSTM-style，应该是 GTU：
$$h_i^l=tanh(XW+b)\otimes \sigma(XV+c)$$
作者将两者进行了对比，发现 GLU 效果更好。
residual connection: 为了得到更 deep 的卷积神经网络，作者增加了残差链接。
For instance, stacking 6 blocks with k = 5 results in an input field of 25 elements, i.e. each output depends on 25 inputs. Non-linearities allow the networks to exploit the full input field, or to focus on fewer elements if needed.
For encoder networks we ensure that the output of the convolutional layers matches the input length by padding the input at each layer. However, for decoder networks we have to take care that no future information is available to the decoder (Oord et al., 2016a). Specifically, we pad the input by k − 1 elements on both the left and right side by zero vectors, and then remove k elements from the end of the convolution output.
得到 attention vector 跟 rnn-decoder 有所不同，这里加上了 input element embedding $e_j$.
至于这里为什么要加 $e_j$?
We found adding e_j to be beneficial and it resembles key-value memory networks where the keys are the z_j^u and the values are the z^u_j + e_j (Miller et al., 2016). Encoder outputs z_j^u represent potentially large input contexts and e_j provides point information about a specific input element that is useful when making a prediction. Once c^l_i has been computed, it is simply added to the output of the corresponding decoder layer h^l_i.
This can be seen as attention with multiple ’hops’ (Sukhbaatar et al., 2015) compared to single step attention (Bahdanau et al., 2014; Luong et al., 2015; Zhou et al., 2016; Wu et al., 2016). In particular, the attention of the first layer determines a useful source context which is then fed to the second layer that takes this information into account when computing attention etc. The decoder also has immediate access to the attention history of the k − 1 previous time steps because the conditional inputs $c^{l-1}_{i−k}, . . . , c^{l-1}i$ are part of $h^{l-1}{i-k}, . . . , h^{l-1}_i$ which are input to $h^l_i$. This makes it easier for the model to take into account which previous inputs have been attended to already compared to recurrent nets where this information is in the recurrent state and needs to survive several non-linearities. Overall, our attention mechanism considers which words we previously attended to (Yang et al., 2016) and performs multiple attention ’hops’ per time step. In Appendix §C, we plot attention scores for a deep decoder and show that at different layers, different portions of the source are attended to.
这个跟 memory networks 中的 multi-hop 是有点类似。
FAST READING COMPREHENSION WITH CONVNETS
Gated Linear Dilated Residual Network (GLDR):
a combination of residual networks (He et al., 2016), dilated convolutions (Yu & Koltun, 2016) and gated linear units (Dauphin et al., 2017).
Repeated dilated convolution (Yu & Koltun, 2016) increases the receptive region of ConvNet outputs exponentially with respect to the network depth, which results in drastically shortened computation paths.
能够显著缩短两个词之间的计算路径。
作者将 GLDR 和 self-attention,以及 RNN 进行了对比，input sequence length n, network width w, kernel size k, and network depth D.
The receptive field of this convolutional network grows
exponentially with depth and soon encompasses a long sequence, essentially enabling it to capture
similar long-term dependencies as an actual sequential model.
感受野的尺寸大小指数增加，能够迅速压缩 long sentence,并 capture 长期依赖。
Convolutional BiDAF. In our convolutional version of BiDAF, we replaced all bidirectional LSTMs with GLDRs . We have two 5-layer GLDRs in the contextual layer whose weights are un-tied. In the modeling layer, a 17-layer GLDR with dilation 1, 2, 4, 8, 16 in the first 5 residual blocks is used, which results in a reception region of 65 words. A 3-layer GLDR replaces the bidirectional LSTM in the output layer. For simplicity, we use same-padding and kernel size 3 for all convolutions unless specified. The hidden size of all GLDRs is 100 which is the same as the LSTMs in BiDAF.
batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case.
关于 batch normalisztion.
从 Ng 的课上截来的一张图，全链接层相比卷积层更容易理解点，但形式上是一样的.
样本数量是 m，第 l 层经过激活函数输出是第 l+1 层的输入，其中第 i 个神经元的值:
线性输出： $z_i^l={w_i^l}^Th^l$.
非线性输出： $h_i^{l+1} = a_i^l=f(z_i^l+b_i^l)$
其中 f 是非线性激活函数，$a_i^l$ 是下一层的 summed inputs. 如果 $a_i^l$ 的分布变化较大（change in a highly correlated way）,下一层的权重 $w^{l+1}$ 的梯度也会相应变化很大（反向传播中 $w^{l+1}$ 的梯度就是 $a_i^l$）。
batch normalization requires running averages of the summed input statistics. In feed-forward networks with fixed depth, it is straightforward to store the statistics separately for each hidden layer. However, the summed inputs to the recurrent neurons in a recurrent neural network (RNN) often vary with the length of the sequence so applying batch normalization to RNNs appears to require different statistics for different time-steps.
In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case.
所以作者在这篇 paper 中提出了 Layer Normalization. 在单个样本上计算均值和方差进行归一化。然而是怎么进行的呢？
Layer Normalization
layer normalization 并不是在样本上求平均值和方差，而是在 hidden units 上求平均值和方差。
RNN is common among the NLP tasks to have different sentence lengths for different training cases. This is easy to deal with in an RNN because the same weights are used at every time-step. But when we apply batch normalization to an RNN in the obvious way, we need to to compute and store separate statistics for each time step in a sequence. This is problematic if a test sequence is longer than any of the training sequences. Layer normalization does not have such problem because its normalization terms depend only on the summed inputs to a layer at the current time-step. It also has only one set of gain and bias parameters shared over all time-steps.
这一部分也解释了 BN 不适用于 RNN 的原因，从 test sequence longer 的角度。RNN 的每个时间步计算共享参数权重.
"""LSTM unit with layer normalization and recurrent dropout. This class adds layer normalization and recurrent dropout to a basic LSTM unit. Layer normalization implementation is based on: https://arxiv.org/abs/1607.06450. "Layer Normalization" Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton and is applied before the internal nonlinearities. Recurrent dropout is base on: https://arxiv.org/abs/1603.05118 "Recurrent Dropout without Memory Loss" Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth. """
"""Initializes the basic LSTM cell. Args: num_units: int, The number of units in the LSTM cell. forget_bias: float, The bias added to forget gates (see above). input_size: Deprecated and unused. activation: Activation function of the inner states. layer_norm: If `True`, layer normalization will be applied. norm_gain: float, The layer normalization gain initial value. If `layer_norm` has been set to `False`, this argument will be ignored. norm_shift: float, The layer normalization shift initial value. If `layer_norm` has been set to `False`, this argument will be ignored. dropout_keep_prob: unit Tensor or float between 0 and 1 representing the recurrent dropout probability value. If float and 1.0, no dropout will be applied. dropout_prob_seed: (optional) integer, the randomness seed. reuse: (optional) Python boolean describing whether to reuse variables in an existing scope. If not `True`, and the existing scope already has the given variables, an error is raised. """
A language model is formalized as a probability distribution over a sequence of strings (words), and traditional methods usually involve making an n-th order Markov assumption and estimating n-gram probabilities via counting and subsequent smoothing (Chen and Goodman 1998). The count-based models are simple to train, but probabilities of rare n-grams can be poorly estimated due to data sparsity (despite smoothing techniques).
While NLMs have been shown to outperform count-based n-gram language models (Mikolov et al. 2011), they are blind to subword information (e.g. morphemes). For example, they do not know, a priori, that eventful, eventfully, uneventful, and uneventfully should have structurally related embeddings in the vector space. Embeddings of rare words can thus be poorly estimated, leading to high perplexities for rare words (and words surrounding them). This is especially problematic in morphologically rich languages with long-tailed frequency distributions or domains with dynamic vocabularies (e.g. social media).
neural language models 将词嵌入到低维的向量中，使得语义相似的词在向量空间的位置也是相近的。然后 Mikolov word2vec 这种方式不能有效的解决子单词的信息问题，比如一个单词的各种形态，也不能认识前缀。这种情况下，不可避免的会造成不常见词的向量表示估计很差，对于不常见词会有较高的困惑度。这对于词语形态很丰富的语言是一个难题，同样这种问题也是动态词表的问题所在（比如社交媒体）。
The final model uses L = 2 biLSTM layers with 4096 units and 512 dimension projections and a residual connection from the first to second
layer. The context insensitive type representation uses 2048 character n-gram convolutional filters followed by two highway layers and a linear projection down to a 512 representation.
Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models.
encoder 编码方式仅仅由 卷积 和 自注意力 机制构成，没了 rnn 速度就是快。
The key motivation behind the design of our model is the following: convolution captures the local structure of the text, while the self-attention learns the global interaction between each pair of words.
we propose a complementary data augmentation technique to enhance the training data. This technique paraphrases the examples by translating the original sentences from English to another language and then back to English, which not only enhances the number of training instances but also diversifies the phrasing.
the combination of convolutions and self-attention is novel, and is significantly better than self-attention alone and gives 2.7 F1 gain in our experiments. The use of convolutions also allows us to take advantage of common regularization methods in ConvNets such as stochastic depth (layer dropout) (Huang et al., 2016), which gives an additional gain of 0.2 F1 in our experiments.
obtain the embedding of each word w by concatenating its word embedding and character embedding.
由词向量和字符向量拼接而成。其中词向量采用预训练的词向量 Glove，并且不可训练，fixed. 只有 OOV (out of vocabulary) 是可训练的，用来映射所有不在词表内的词。
Each character is represented as a trainable vector of dimension p2 = 200, meaning each word can be viewed as the concatenation of the embedding vectors for each of its characters. The length of each word is either truncated or padded to 16. We take maximum value of each row of this matrix to get a fixed-size vector representation of each word.
The encoder layer is a stack of the following basic building block: [convolution-layer × # + self-attention-layer + feed-forward-layer]
其中：
convolution: 使用 depthwise separable convolutions 而不是用传统的 convolution，因为作者发现 it is memory efficient and has better generalization. 怎么理解这个，还得看原 paper. The kernel size is 7, the number of filters is d = 128.
Each of these basic operations (conv/self-attention/ffn) is placed inside a residual block, shown lower-right in Figure 1. For an input x and a given operation f, the output is f(layernorm(x))+x.
We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence.
Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence [1] and Neural Turing Machines [2], because the number of target classes in each step of the output depends on the length of the input, which is variable.
Problems such as sorting variable sized sequences, and various combinatorial optimization problems belong to this class.
和这类问题类似的还有给不定长序列的排序，组合优化等问题。
It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output.
We show Ptr-Nets can be used to learn approximate solutions to three challenging geometric problems – finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem – using training examples
alone.
Ptr-Net 可以用来学习类似的三个几何问题。
Ptr-Nets not only improve over sequence-to-sequence with input attention, but also allow us to generalize to variable size output dictionaries. We show that the learnt models generalize beyond the maximum lengths they were trained on.
Ptr-Net 不仅可以提升 seq2seq with attention,而且能够泛化到变化的 dictionayies.
In this sequence-to-sequence model, the output dictionary size for all symbols $C_i$ is fixed and equal to n, since the outputs are chosen from the input. Thus, we need to train a separate model for each n. This prevents us from learning solutions to problems that have an output dictionary with a size that depends on the input sequence length.
在每一个 decoder step，先计算 $e_{ij}$ 得到对齐概率(或者说 how well input position j matches output position i)，然后做一个 softmax 得到 $a_{ij}$，再对 $a_{ij}$ 做一个加权和作为 context vector $c_i$，得到这个 context vector 之后在固定大小的 output dictionary 上做 softmax 预测输出的下一个单词。
This model performs significantly better than the sequence-to-sequence model on the convex hull problem, but it is not applicable to problems where the output dictionary size depends on the input.
Nevertheless, a very simple extension (or rather reduction) of the model allows us to do this easily.
We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation.
From a cognitive perspective, the copying mechanism is related to rote memorization, requiring less understanding but ensuring high literal fidelity. 从认知学角度，copy机制近似于死记硬背，不需要太多的理解，但是要保证文字的保真度。
From a modeling perspective, the copying operations are more rigid and symbolic, making it more difficult than soft attention mechanism to integrate into a fully differentiable neural model. 从模型的角度，copy 操作更加死板和符号化，这也使得相比 soft attention 机制更难整合到一个完整的可微分的神经模型中去。
Prediction: COPYNET predicts words based on a mixed probabilistic model of two modes, namely the generate-mode and the copymode, where the latter picks words from the source sequence. 下一个词的预测由两种模式混合而成。生成 generate-mode 和 copy-mode. 后者就像前面 Ptr-Net 所说的，在 source sentence 获取词。
State Update: the predicted word at time t−1 is used in updating the state at t, but COPYNET uses not only its word-embedding but also its corresponding location-specific hidden state in M (if any). 更新 decoder 中的隐藏状态时，t 时间步的隐藏状态不仅与 t-1 步生成词的 embedding vector 有关，还与这个词对应于 source sentence 中的隐藏状态的位置有关。
Reading M: in addition to the attentive read to M, COPYNET also has“selective read” to M, which leads to a powerful hybrid of
content-based addressing and location-based addressing. 什么时候需要 copy，什么时候依赖理解来回答，怎么混合这两种模式很重要。
The architecture is based on match-LSTM, a model we proposed
previously for textual entailment, and Pointer Net, a sequence-to-sequence model proposed by Vinyals et al. (2015) to constrain the output tokens to be from the input sequences.
MCTest: A challenge dataset for the open-domain machine comprehension of text.
Teaching machines to read and comprehend.
The Goldilocks principle: Reading children’s books with explicit memory representations.
Towards AI-complete question answering: A set of prerequisite toy tasks.
SQuAD: 100,000+ questions for machine comprehension of text.
SQuAD
Traditional solutions to this kind of question answering tasks rely on NLP pipelines that involve multiple steps of linguistic analyses and feature engineering, including syntactic parsing, named entity recognition, question classification, semantic parsing, etc. Recently, with the advances of applying neural network models in NLP, there has been much interest in building end-to-end neural architectures for various NLP tasks, including several pieces of work on machine comprehension.
The Goldilocks principle: Reading children’s books with explicit memory representations.
Attention-based convolutional neural network for machine comprehension
Text understanding with the attention sum reader network.
Consensus attention-based neural networks for chinese reading comprehension.
However, given the properties of previous machine comprehension datasets, existing end-to-end neural architectures for the task either rely on the candidate answers (Hill et al., 2016; Yin et al., 2016) or assume that the answer is a single token (Hermann et al., 2015; Kadlec et al., 2016; Cui et al., 2016), which make these methods unsuitable for the SQuAD dataset.
We propose two ways to apply the Ptr-Net model for our task: a sequence model and a boundary model. We also further extend the boundary model with a search mechanism.
作者提出的两种模型。
Model Architecture
Match-LSTM
Pointer Network
Pointer Network (Ptr-Net) model : to solve a special kind of problems where we want to generate an output sequence whose tokens must come from the input sequence. Instead of picking an output token from a fixed vocabulary, Ptr-Net uses attention mechanism as a pointer to select a position from the input sequence as an output symbol.
An LSTM preprocessing layer that preprocesses the passage and the question using LSTMs. 使用 LSTM 处理 question 和 passage.
A match-LSTM layer that tries to match the passage against the question. 使用 match-LSTM 对lstm编码后的 question 和 passage 进行匹配。
An Answer Pointer (Ans-Ptr) layer that uses Ptr-Net to select a set of tokens from the passage as the answer. The difference between the two models only lies in the third layer. 使用 Pointer 来选择 tokens.
So the main difference from the sequence model above is that in the boundary model we do not need to add the zero padding to Hr, and the probability of generating an answer is simply modeled as:
$$p(a|H^r)=p(a_s|H^r)p(a_e|a_s, H^r)$$
Search mechanism, and bi-directional Ans-Ptr.
Training
Dataset
SQuAD: Passages in SQuAD come from 536 articles from Wikipedia covering a wide range of topics. Each passage is a single paragraph from a Wikipedia article, and each passage has around 5 questions associated with it. In total, there are 23,215 passages and 107,785 questions. The data has been split into a training set (with 87,599 question-answer pairs), a development set (with 10,570 questionanswer pairs) and a hidden test set
configuration
dimension l of the hidden layers is set to 150 or 300.
Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query.
机器阅读的定义，query 和 context 之间的交互。
Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention.
传统的使用 attention 机制的方法。
In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bidirectional attention flow mechanism to obtain a query-aware context representation without early summarization.
Attention mechanisms in previous works typically have one or more of the following characteristics. First, the computed attention weights are often used to extract the most relevant information from the context for answering the question by summarizing the context into a fixed-size vector. Second, in the text domain, they are often temporally dynamic, whereby the attention weights at the current time step are a function of the attended vector at the previous time step. Third, they are usually uni-directional, wherein the query attends on the context paragraph or the image.
First, our attention layer is not used to summarize the context paragraph into a fixed-size vector. Instead, the attention is computed for every time step, and the attended vector at each time step, along with the representations from previous layers, is allowed to flow through to the subsequent modeling layer. This reduces the information loss caused by early summarization.
Second, we use a memory-less attention mechanism. That is, while we iteratively compute attention through time as in Bahdanau et al. (2015), the attention at each time step is a function of only the query and the context paragraph at the current time step and does not directly depend on the attention at the previous time step.
We hypothesize that this simplification leads to the division of labor between the attention layer and the modeling layer. It forces the attention layer to focus on learning the attention between the query and the context, and enables the modeling layer to focus on learning the interaction within the query-aware context representation (the output of the attention layer). It also allows the attention at each time step to be unaffected from incorrect attendances at previous time steps.
Character Embedding Layer and Word Embedding Layer -> Contextual Embedding Layer -> Attention Flow Layer -> Modeling Layer -> Output Layer
Character Embedding Layer and word embedding alyer
charatter embedding of each word using CNN.The outputs of the CNN are max-pooled over the entire width to obtain a fixed-size vector for each word.
pre-trained word vectors, GloVe
concatenation of them above and is passed to a two-layer highway networks.
context -> $X\in R^{d\times T}$
query -> $Q\in R^{d\times J}$
contextual embedding layer
model the temporal interactions between words using biLSTM.
context -> $H\in R^{2d\times T}$
query -> $U\in R^{2d\times J}$
前三层网络是在不同的粒度层面来提取 context 和 query 的特征。
attention flow layer
the attention flow layer is not used to summarize the query and context into single feature vectors. Instead, the attention vector at each time step, along with the embeddings from previous layers, are allowed to flow through to the subsequent modeling layer.
输入是 H 和 G，输出是 query-aware vector G, 以及上一层的 contextual layer.
The answer span $(k; l)$ where $k \le l$ with the maximum value of $p^1_kp^2_l$ is chosen, which can be computed in linear time with dynamic programming.
Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift.
stochastic gradient is simple and effective, it requires careful tuning of the model hyper-parameters, specifically the learning rate and the initial parameter values. The training is complicated by the fact that the inputs to each layer are affected by the parameters of all preceding layers – so that small changes to the network parameters amplify as the network becomes deeper.
The change in the distributions of layers’ inputs presents a problem because the layers need to continuously adapt to the new distribution. When the input distribution to a learning system changes, it is said to experience covariate shift (Shimodaira, 2000). This is typically handled via domain adaptation (Jiang, 2008). However, the notion of covariate shift can be extended beyond the learning system
as a whole, to apply to its parts, such as a sub-network or a layer.
Therefore, the input distribution properties that aid the network generalization – such as having the same distribution between the training and test data – apply to training the sub-network as well.As such it is advantageous for the distribution of x to remain fixed over time.
有助于网络泛化的输入分布属性：例如在训练和测试数据之间具有相同的分布，也适用于训练子网络
Fixed distribution of inputs to a sub-network would have positive consequences for the layers outside the subnetwork, as well.
其中 u 是输入， g 是 sigmoid 激活函数 $g(x)=\dfrac{1}{1+exp(x)}$，当 |x| 增加时，$g’(x)$ 趋近于0, 这意味着 $x=Wu+b$ 的所有维度，除了绝对值较小的维度，其他的流向输入 u 的梯度都会消失,也就是进入非线性的饱和区域，这会降低模型训练速度。
在实际应用中，对于非线性饱和的情况，已经有很有对应策略：
ReLU
初始化 Xavier initialization.
用一个较小的学习速率进行学习
If, however, we could ensure that the distribution of nonlinearity inputs remains more stable as the network trains, then the optimizer would be less likely to get stuck in the saturated regime, and the training would accelerate.
如果保证非线性输入的分布稳定，优化器也就不会陷于饱和区域了，训练也会加速。
We refer to the change in the distributions of internal nodes of a deep network, in the course of training, as Internal Covariate Shift. Eliminating it offers a promise of faster training. We propose a new mechanism, which we call Batch Normalization, that takes a step towards reducing internal covariate shift, and in doing so dramatically accelerates the training of deep neural nets. It accomplishes this via a normalization step that fixes the means and variances of layer inputs.
Batch Normalization also has a beneficial effect on the gradient flow through the network, by reducing the dependence of gradients on the scale of the parameters or
of their initial values. This allows us to use much higher learning rates without the risk of divergence. Furthermore, batch normalization regularizes the model and reduces the need for Dropout (Srivastava et al., 2014). Finally, Batch Normalization makes it possible to use saturating nonlinearities by preventing the network from getting stuck in the saturated modes.
It has been long known (LeCun et al., 1998b; Wiesler & Ney, 2011) that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated.
However, if these modifications are interspersed with the optimization steps, then the gradient descent step may attempt to update the parameters in a way that requires the normalization to be updated, which reduces the effect of the gradient step.
This problem can get worse if the normalization not only centers but also scales the activations. We have observed this empirically in initial experiments, where the model blows up when the normalization parameters are computed outside the gradient descent step.
The issue with the above approach is that the gradient descent optimization does not take into account the fact that the normalization takes place. To address this issue, we would like to ensure that, for any parameter values, the network always produces activations with the desired distribution.Doing so would allow the gradient of the loss with respect to the model parameters to account for the normalization, and for its dependence on the model parameters Θ.
Since the full whitening of each layer’s inputs is costly, we make two necessary simplifications. The first is that instead of whitening the features in layer inputs and outputs jointly, we will normalize each scalar feature independently, by making it have zero mean and unit variance.
where the expectation and variance are computed over the
training data set.
其中均值和方差是基于整个训练集计算得到的。
Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity.
In the batch setting where each training step is based on the entire training set, we would use the whole set to normalize activations. However, this is impractical when using stochastic optimization. Therefore, we make the second simplification: since we use mini-batches in stochastic gradient training, each mini-batch produces estimates of the mean and variance of each activation.
Note that the use of mini-batches is enabled by computation of per-dimension variances rather than joint covariances; in the joint case, regularization would be required since the mini-batch size is likely to be smaller than the number of activations being whitened, resulting in singular covariance matrices.
Thus, BN transform is a differentiable transformation that introduces normalized activations into the network. This ensures that as the model is training, layers can continue learning on input distributions that exhibit less internal covariate shift, thus accelerating the training.
Training and Inference with Batch-Normalized Networks
The normalization of activations that depends on the mini-batch allows efficient training, but is neither necessary nor desirable during inference; we want
the output to depend only on the input, deterministically. For this, once the network has been trained, we use the normalization
$\hat x = \dfrac{x-E[x]}{\sqrt{Var[x]+\epsilon}}$
using the population, rather than mini-batch, statistics.
Using moving averages instead, we can track the accuracy of a model as it trains. Since the means and variances are fixed during inference, the normalization is simply a linear transform applied to each activation.
In traditional deep networks, too high a learning rate may result in the gradients that explode or vanish, as well as getting stuck in poor local minima.
学习率过大容易发生梯度消失和梯度爆炸，从而陷入局部最小值。
By normalizing activations throughout the network, it prevents small changes in layer parameters from amplifying as the data propagates through a deep network.
通过规范化整个网络中的激活，可以防止层参数的微小变化在数据通过深层网络传播时放大。
Batch Normalization also makes training more resilient to the parameter scale. Normally, large learning rates may increase the scale of layer parameters, which then amplify the gradient during backpropagation and lead to the model explosion. However, with Batch Normalization, backpropagation through a layer is unaffected by the scale of its parameters.
Graves, Alex, et al. “A novel connectionist system for unconstrained handwriting recognition.” IEEE transactions on pattern analysis and machine intelligence 31.5 (2009): 855-868.
It may work better, especially in recurrent networks (Hinton)
对网络的响应加噪声
如在前向传播过程中，让默写神经元的输出变为 binary 或 random。显然，这种有点乱来的做法会打乱网络的训练过程，让训练更慢，但据 Hinton 说，在测试集上效果会有显著提升 （But it does significantly better on the test set!）。
结合多种模型
简而言之，训练多个模型，以每个模型的平均输出作为结果。
从 N 个模型里随机选择一个作为输出的期望误差 ，会比所有模型的平均输出的误差 大（我不知道公式里的圆括号为什么显示不了）：
其中 attention distribution 是由 content-based attention 生成的。具体过程还是看英文描述吧：
The attending RNN generates a query describing what it wants to focus on. Each item is dot-producted with the query to produce a score, describing how well it matches the query. The scores are fed into a softmax to create the attention distribution.
Then an RNN runs, generating a description of the image. As it generates each word in the description, the RNN focuses on the conv net’s interpretation of the relevant parts of the image. We can explicitly visualize this:
More broadly, attentional interfaces can be used whenever one wants to interface with a neural network that has a repeating structure in its output.
Adaptive Computation time
Standard RNNs do the same amount of computation for each time step. This seems unintuitive. Surely, one should think more when things are hard? It also limits RNNs to doing O(n) operations for a list of length n.
标准的 RNN 在每一个时间步都做着相同的大量的计算。它貌似是很合理的，但当处理的信息很复杂时，是否可以在一个时间步考虑更多？这样同样的计算量的方式限制了 RNN 在处理长度为 n 的序列时，其复杂度也为 O(n).
Adaptive Computation Time [15] is a way for RNNs to do different amounts of computation each step. The big picture idea is simple: allow the RNN to do multiple steps of computation for each time step.
In order for the network to learn how many steps to do, we want the number of steps to be differentiable. We achieve this with the same trick we used before: instead of deciding to run for a discrete number of steps, we have an attention distribution over the number of steps to run. The output is a weighted combination of the outputs of each step.
There are a few more details, which were left out in the previous diagram. Here’s a complete diagram of a time step with three computation steps.
上图中还有更多的细节。下图是 RNN 中的在一个时间步中有3个步数的计算量。
That’s a bit complicated, so let’s work through it step by step. At a high-level, we’re still running the RNN and outputting a weighted combination of the states:
The weight for each step is determined by a “halting neuron.” It’s a sigmoid neuron that looks at the RNN state and gives a halting weight, which we can think of as the probability that we should stop at that step.
We have a total budget for the halting weights of 1, so we track that budget along the top. When it gets to less than epsilon, we stop
我们设总的停止权重为 1,依次减去每步输出的 halt 值，直到剩余的 halt 值小于 epsilon 后就停止。
When we stop, might have some left over halting budget because we stop when it gets to less than epsilon. What should we do with it? Technically, it’s being given to future steps but we don’t want to compute those, so we attribute it to the last step.
When training Adaptive Computation Time models, one adds a “ponder cost” term to the cost function. This penalizes the model for the amount of computation it uses. The bigger you make this term, the more it will trade-off performance for lowering compute time.
However, their memory (encoded by hidden states and weights) is typically too small, and is not compartmentalized enough to accurately remember facts from the past (knowledge is compressed into dense vectors). RNNs are known to have difficulty in performing memorization.
internal state u: $q\stackrel B \longrightarrow u$
2.计算attention，也就是query的向量表示u，和input中各个sentence的向量表示 $m_i$ 的匹配度。compute the match between u and each memory mi by taking the inner product followed by a softmax.
Overall, it is similar to the Memory Network model in [23], except that the hard max operations within each layer have been replaced with a continuous weighting from the softmax.
However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images.
Like the original DMN, this memory network requires that supporting facts are labeled during QA training. End-toend memory networks (Sukhbaatar et al., 2015) do not have this limitation.
Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions.
这篇文章对DMN中的 input module进行了修改，并且提出了新的模型架构适用于图像的。
DMN 存在的两个问题：
输入模块只考虑了过去信息，没考虑到将来信息
只用 word level 的 GRU，很难记忆远距离 supporting sentences 之间的信息。
总的来说这篇文章贡献主要还是在应用到图像上了，至于作者所说的 input module的改进，只是为了减少计算量，而且改进版中的 bi-RNN 和 position encoding 都是在别人的论文中出现了的。
Without global information, their representational power is quite limited, with simple issues like object scaling or locational variance causing accuracy problems. 强调了为什么要使用 input fusion layer.
Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states ht, as a function of the previous hidden state ht−1 and the input for position t.
This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples.
RNN模型有两个很致命的缺点：
$$y_t=f(y_{t-1},x_t)$$
一是无法解决长句子的长期依赖的问题（这是因为反向传播，loss的梯度很难传递到比较靠前的位置，造成前面的词对整体的影响偏小，[Gradient flow in
Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences. In all but a few cases, however, such attention mechanisms are used in conjunction with a recurrent network.
2.1 Extended Neural GPU [16], ByteNet [18] and ConvS2S [9]
2.2 Self-attention
Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.
End-to-end memory networks are based on a recurrent attention mechanism instead of sequence aligned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks.
the encoder maps an input sequence of symbol representations $(x_1,…,x_n)$ to a sequence of continuous representations $z = (z_1,…,z_n)$. Given z, the decoder then generates an output sequence $(y_1,…,y_m)$ of symbols one element at a time. At each step the model is auto-regressive [10], consuming the previously generated symbols as additional input when generating the next.
We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.
目的应该就是让下一个t时刻的生成词只依赖于t时刻之前的词。
Attention
Really love this short description of attention:
An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
Scaled Dot-Product Attention
queries: $Q\in R^{n\times d_k}$
keys: $K\in R^{n\times d_k}$
values: $V\in R^{n\times d_v}$
计算向量內积作为相似度，并使用softmax计算权重，然后加权求和。 这种 attention 其实也很常见了，这里 google 算是给这种结构一个官方的名字吧。
we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{model}$. In our model, we share the same weight matrix between the two embedding layers.
""" :param inputs: A `Tensor` with type `int32` or `int64` containing the ids to be looked up in `lookup table`. shape is [batch, sentence_len] :param vocab_size: vocabulary size :param num_units: Number of embedding hidden units. in the paper, it is called d_model :param zero_pad: If True, all the values of the fist row (id 0) should be constant zeros. :param scale: If True. the outputs is multiplied by sqrt num_units. :param reuse: Boolean, whether to reuse the weights of a previous layer by the same name. :return: A `Tensor` with one more rank than inputs's. The last dimensionality should be `num_units`. """
with tf.variable_scope("embedding-layer", reuse=reuse):
'''Sinusoidal Positional_Encoding. Args: inputs: A 2d Tensor with shape of (N, T). num_units: Output dimensionality zero_pad: Boolean. If True, all the values of the first row (id = 0) should be constant zero scale: Boolean. If True, the output will be multiplied by sqrt num_units(check details from paper) scope: Optional scope for `variable_scope`. reuse: Boolean, whether to reuse the weights of a previous layer by the same name. Returns: A 'Tensor' with one more rank than inputs's, with the dimensionality should be 'num_units' '''
N, T = inputs.get_shape().as_list() # N means batch_size, T means the sentence length.
In RNN (LSTM), the notion of time step is encoded in the sequence as inputs/outputs flow one at a time. In FNN, the positional encoding must be preserved to represent the time in some way to preserve the positional encoding. In case of the Transformer authors propose to encode time as sine wave, as an added extra input. Such signal is added to inputs and outputs to represent time passing.
In general, adding positional encodings to the input embeddings is a quite interesting topic. One way is to embed the absolute position of input elements (as in ConvS2S). However, authors use “sine and cosine functions of different frequencies”. The “sinusoidal” version is quite complicated, while giving similar performance to the absolute position version. The crux is however, that it may allow the model to produce better translation on longer sentences at test time (at least longer than the sentences in the training data). This way sinusoidal method allows the model to extrapolate to longer sequence lengths.
Transformer reduces the number of operations required to relate (especially distant) positions in input and output sequence to a O(1). However, this comes at cost of reduced effective resolution because of averaging attention-weighted positions.
h = 8 attention layers (aka “heads”): that represent linear projection (for the purpose of dimension reduction) of key K and query Q into $d_k$-dimension and value V into $d_v$-dimension:
$$head_i = Attention(Q W^Q_i, K W^K_i, V W^V_i) , i=1,\dots,h$$
where $W^0\in\mathbb{R}^{d_{hd_v}\times d_{model}}$
关于 attention 在模型中的应用，有三种情况
1.In “encoder-decoder attention” layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder。
2.The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder.
3.Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property. We implement this
inside of scaled dot-product attention by masking out (setting to −1) all values in the input of the softmax which correspond to illegal connections.
""" multi scaled dot product attention :param q: A 3d tensor with shape of [batch, length_q, d_k]. :param k: A 3d tensor with shape of [batch, lenght_kv, d_k]. :param v: :param heads:An int. Number of heads. :param dropout_keep_prob: :param causality: If true, units that reference the future are masked. :return: """
# 1. Linear projections
with tf.variable_scope('linear-projection-multiheads'):
""" Point-wise feed forward net. :param inputs: A 3D tensor with shape of [batch, length_q, d_model] :param num_units1: A integers. :param num_units2: A integers :param reuse: Boolean, whether to reuse the weights of a previous layer by the same name. :return: A 3d tensor with the same shape and dtype as inputs """
During training, we employed label smoothing of value ls = 0:1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
deflabel_smoothing(inputs, epsilon=0.1):
""" Applies label smoothing. See https://arxiv.org/abs/1512.00567 :param inputs: A 3d tensor with shape of [N, T, V], where V is the number of vocabulary. :param epsilon: Smoothing rate. For example, """
K = inputs.get_shape().as_list()[-1] # number of channels
In order to avoid neurons becoming too correlated and ending up in poor local minimize, it is often helpful to randomly initialize parameters. 为了避免神经元高度相关和局部最优化，常常需要采用随机初始化权重参数，最常用的就是Xavier initiazation.
""" Args: shape: Tuple or 1-d array that species dimensions of requested tensor. Returns: out: tf.Tensor of specified shape sampled from Xavier distribution. """
epsilon = np.sqrt(6/np.sum(shape))
out = tf.Variable(tf.random_uniform(shape=shape, minval=-epsilon, maxval=epsilon))
To avoid parameters from exploding or becoming highly correlated, it is helpful to augment our cost function with a Gaussian prior: this tends to push parameter weights closer to zero, without constraining their direction, and often leads to classifiers with better generalization ability.
If we maximize log-likelihood (as with the cross-entropy loss, above), then the Gaussian prior becomes a quadratic term 1 (L2 regularization):
