text: We propose a large margin criterion for training neural language models. Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. However, we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation. The proposed method aims to enlarge the margin between the "good" and "bad" sentences in a task-specific sense. It is trained end-to-end and can be widely applied to tasks that involve re-scoring of generated text. Compared with minimum-PPL training, our method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation.
  @cite_10: This paper describes discriminative language modeling for a large vocabulary speech recognition task. We contrast two parameter estimation methods: the perceptron algorithm, and a method based on maximizing the regularized conditional log-likelihood. The models are encoded as deterministic weighted finite state automata, and are applied by intersecting the automata with word-lattices that are the output from a baseline recognizer. The perceptron algorithm has the benefit of automatically selecting a relatively small feature set in just a couple of passes over the training data. We describe a method based on regularized likelihood that makes use of the feature set given by the perceptron algorithm, and initialization with the perceptron's weights; this method gives an additional 0.5 reduction in word error rate (WER) over training with the perceptron alone. The final system achieves a 1.8 absolute reduction in WER for a baseline first-pass recognition system (from 39.2 to 37.4 ), and a 0.9 absolute reduction in WER for a multi-pass recognition system (from 28.9 to 28.0 ).
summary: While these works all adopt PPL as the metric to optimize, sometimes one may optimize a task-specific objective. For example, and propose discriminative LMs to improve speech recognition. The common methodology therein is to fit a probabilistic model, e.g., conditional random field @cite_10 , to the space of text candidates, and maximize the probability at the desired candidate. The problem is often solved by perceptron algorithm. However, these methods all rely on ad-hoc choice of features, e.g., counts of @math -grams where @math varies in a small range (e.g.,1 to 3). Moreover, it is also not clear how these methods would take advantage of an existing language model (trained on large unsupervised corpus). Nevertheless, the same methodology can be extended to RNNLMs, thus avoiding the aforementioned limitations. For example, train an RNNLM by favoring sentences with high BLEU scores and integrate it into a phrase-based MT decoder.