  To encourage intra-class compactness and inter-class separability among
trainable feature vectors, large-margin softmax methods are developed and
widely applied in the face recognition community. The introduction of the
large-margin concept into the softmax is reported to have good properties such
as enhanced discriminative power, less overfitting and well-defined geometric
intuitions. Nowadays, language modeling is commonly approached with neural
networks using softmax and cross entropy. In this work, we are curious to see
if introducing large-margins to neural language models would improve the
perplexity and consequently word error rate in automatic speech recognition.
Specifically, we first implement and test various types of conventional margins
following the previous works in face recognition. To address the distribution
of natural language data, we then compare different strategies for word vector
norm-scaling. After that, we apply the best norm-scaling setup in combination
with various margins and conduct neural language models rescoring experiments
in automatic speech recognition. We find that although perplexity is slightly
deteriorated, neural language models with large-margin softmax can yield word
error rate similar to that of the standard softmax baseline. Finally, expected
margins are analyzed through visualization of word vectors, showing that the
syntactic and semantic relationships are also preserved.
