# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# Laplace smoothing (add-one smoothing) : $ P(w_i| w_{i-1}) =  {c(w_{i-1} w_i)}  {c(w_{i-1}) } $  
# Additive smoothing (add-n smoothing) :  
# Good-Turing smoothing : $ r^{*} = (r+1) \frac{n_{r+1}}{n_r} $  
# $N = \sum\limits_{r = 1, ..., + \infty} r* n_r $  
# $P(w_i | w_{i-n+1}^{i-1}) = \frac{r^{*}}{N}$

# Jelinek-Mercer smoothing :  $ P_{interp} (w_i| w_{i-n+1}^{i-1})  = \lambda_{w_{i-n+1}^{i-1}} P_{ML} (w_i| w_{i-n+1}^{i-1})   +   (1-\lambda_{w_{i-n+1}^{i-1}})P_{interp} (w_i| w_{i-n+2}^{i-1}) $ 高阶和低阶模型的插值(interpolation)

# Witten-Bell smoothing : $ 1 - \lambda_{WB} = \frac { N_{word}(w_{i-n+1}^{i-1})  }{N_{word}(w_{i-n+1}^{i-1}) + \sum_{w_i} c(w_{i-n+1}^i) } $ 代表了 $ w_{i-n+1}...w{i-1}$后词的丰富程度,  
# 其中$ N_{word}(w_{j}^{i-1}) = | \{ w_i | c(w_{j}^{i-1}w_i) >0  \}|$代表了$w_{j}^{i-1}$后不同词的数量

# Kneser-Ney smoothing : $ P(w_i | w_{i-1}) = \frac { max( c(w_{i-1} w_i)-d,0)} { \sum\limits_{w_j} c( w_{i-1}  w_j) }  + \frac {  |  \{w_i : c(w_{i-1} w_i)>0 \}| }{ \sum\limits_{w_j} c( w_{i-1}  w_j) } * P_{continuation}(w_i)$  
# $  P_{continuation}(w_i) =  \frac{  | \{ w_j : c(w_j w_i) > 0 \}|} { \sum _{w_i} | \{ w_j : c(w_j w_i) > 0 \}| } $  表示 $w_i$上下文的丰富程度,越丰富,概率越大



StarcoderdataJupyterScriptsDedupFiltered