  Lifelong learning capabilities are crucial for sentiment classifiers to
process continuous streams of opinioned information on the Web. However,
performing lifelong learning is non-trivial for deep neural networks as
continually training of incrementally available information inevitably results
in catastrophic forgetting or interference. In this paper, we propose a novel
iterative network pruning with uncertainty regularization method for lifelong
sentiment classification (IPRLS), which leverages the principles of network
pruning and weight regularization. By performing network pruning with
uncertainty regularization in an iterative manner, IPRLS can adapta single BERT
model to work with continuously arriving data from multiple domains while
avoiding catastrophic forgetting and interference. Specifically, we leverage an
iterative pruning method to remove redundant parameters in large deep networks
so that the freed-up space can then be employed to learn new tasks, tackling
the catastrophic forgetting problem. Instead of keeping the old-tasks fixed
when learning new tasks, we also use an uncertainty regularization based on the
Bayesian online learning framework to constrain the update of old tasks weights
in BERT, which enables positive backward transfer, i.e. learning new tasks
improves performance on past tasks while protecting old knowledge from being
lost. In addition, we propose a task-specific low-dimensional residual function
in parallel to each layer of BERT, which makes IPRLS less prone to losing the
knowledge saved in the base BERT network when learning a new task. Extensive
experiments on 16 popular review corpora demonstrate that the proposed IPRLS
method sig-nificantly outperforms the strong baselines for lifelong sentiment
classification. For reproducibility, we submit the code and data
at:https://github.com/siat-nlp/IPRLS.
