  Lip reading aims to predict speech based on lip movements alone. As it
focuses on visual information to model the speech, its performance is
inherently sensitive to personal lip appearances and movements. This makes the
lip reading models show degraded performance when they are applied to unseen
speakers due to the mismatch between training and testing conditions. Speaker
adaptation technique aims to reduce this mismatch between train and test
speakers, thus guiding a trained model to focus on modeling the speech content
without being intervened by the speaker variations. In contrast to the efforts
made in audio-based speech recognition for decades, the speaker adaptation
methods have not well been studied in lip reading. In this paper, to remedy the
performance degradation of lip reading model on unseen speakers, we propose a
speaker-adaptive lip reading method, namely user-dependent padding. The
user-dependent padding is a speaker-specific input that can participate in the
visual feature extraction stage of a pre-trained lip reading model. Therefore,
the lip appearances and movements information of different speakers can be
considered during the visual feature encoding, adaptively for individual
speakers. Moreover, the proposed method does not need 1) any additional layers,
2) to modify the learned weights of the pre-trained model, and 3) the speaker
label of train data used during pre-train. It can directly adapt to unseen
speakers by learning the user-dependent padding only, in a supervised or
unsupervised manner. Finally, to alleviate the speaker information
insufficiency in public lip reading databases, we label the speaker of a
well-known audio-visual database, LRW, and design an unseen-speaker lip reading
scenario named LRW-ID.
