<jupyter_start><jupyter_text>Introduction to Python and Natural Language Technologies__Lecture 9, Transformers, BERT____April 13, 2021____<NAME>__<jupyter_code>import gc
from IPython.display import Image
import numpy as np
import seaborn as sns
import torch
import torch.nn as nn

from transformers import pipeline
from transformers import AutoTokenizer, AutoModel<jupyter_output><empty_output><jupyter_text>Attention mechanismAttention:- emphasizes the important part of the input- and de-emphasizes the rest.- Mimics cognitive attention.Method:- It does this by assigning weights to the elements of the input sequence.- The weights depend on the current context in the decoder:    - the current decoder hidden state,    - the previous output.- The source vectors are multiplied by the weights and then summed -> **context vector**- The context vector is used for predicting the next output symbol.<jupyter_code>Image("img/dl/attention_mechanism.jpg")<jupyter_output><empty_output><jupyter_text>ProblemsRecall that we used recurrent neural cells, specifically LSTMs to encode and decode sequences.__Problem 1. No parallelism__LSTMs are recurrent, they rely on their left and right history (horizontal arrows), so the symbols need to be processed in order -> no parallelism.__Problem 2. Long-range dependencies__Long-range dependencies are not infrequent in NLP."The **people/person** who called and wanted to rent your house when you go away next year **are/is** from California" -- Miller & Chomsky 1963LSTMs have a problem capturing these because there are too many backpropagation steps between the symbols.  TransformersIntroduced in [Attention Is All You Need](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) by Vaswani et al., 2017Transformers solve Problem 1 by relying purely on attention instead of recurrence.Not having recurrent connections means that sequence position no longer matters.Recurrence is replaced by **self attention**.Each symbol is encoded the following way:__Step 1__: the encoder 'looks' at the other symbols in the input sequence    - In the example above: the representation of **are/is** depends on **people/person** more than any other word in the sentence, it should receive the highest attention weight.<jupyter_code>Image("http://jalammar.github.io/images/t/transformer_self-attention_visualization.png", embed=True)  # from Illustrated Transformers<jupyter_output><empty_output><jupyter_text>__Step 2__: the context vector is passed through a feed-forward network which is shared across all symbols.<jupyter_code>Image("http://jalammar.github.io/images/t/encoder_with_tensors.png", embed=True)  # from Illustrated Transformers<jupyter_output><empty_output><jupyter_text>This visualization is available in the [Tensor2tensor notebook in Google Colab](https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb)  Other components__Residual connections__- Also called __skip connections__- The output of a module is added to the input$$\text{output} = \text{layer}(\text{input}) + \text{input}$$__Softmax__- Only used in the decoder- Maps the output vector to a probability distribution    - In other words it tells us how likely each symbol is. Multiple heads and layersTransformers have a number of additional components summarized in this figure:<jupyter_code>Image("img/dl/transformer.png")  # from Vaswani et al. 2018<jupyter_output><empty_output><jupyter_text>PyTorch supportPyTorch has a `nn.Transformer` class and its encoder and decoder versions.<jupyter_code>from torch.nn import TransformerEncoder, TransformerEncoderLayer
embedding_dim = 12
# num_heads = 5  # embedding_dim must be divisible by the number of heads
num_heads = 2
hidden_size = 7
dropout = 0.2
TransformerEncoderLayer(embedding_dim, num_heads, hidden_size, dropout)
layer = TransformerEncoderLayer(embedding_dim, num_heads, hidden_size, dropout)
TransformerEncoder(layer, 2)
encoder = TransformerEncoder(layer, 2)

sequence_len = 9
batch_size = 3
X = torch.rand((sequence_len, batch_size, embedding_dim))
y = encoder(X)
y.size()<jupyter_output><empty_output><jupyter_text>Positional encodingWithout recurrence word order information is lost.Positional information is important:    John loves Mary.    Mary loves John.Transformers apply positional encoding:$$\text{PE}_{\text{pos},2i} = \sin(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}), \\\text{PE}_{\text{pos},2i+1} = \cos(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}),$$where:- $d_{\text{model}}$ is the input dimension to the Transformer, usually the embedding size- $\text{pos}$ is the position of the symbol in the input sequence i.e. first word, second word etc.- $i$ is the coordinate index in the input vector. Let's create a position encoder in PyTorch.For $\text{pos}=0$, the sine values are 0, and the cosine values are 1:<jupyter_code>t = torch.FloatTensor([0.])
torch.cos(t), torch.sin(t)<jupyter_output><empty_output><jupyter_text>For large $i$ values, the denominator is close to 10000, so it's again close to 0.<jupyter_code># Pick a few random values for pos
pos = torch.randint(512, size=(10, ))
print(pos)

# Divide by 10000^2*i/d_model. Make 2*i/d_model close to one (high 2*i values)
t = pos / (10000 ** 0.95)
print(t)
torch.cos(t), torch.sin(t)<jupyter_output>tensor([445, 109, 251,  47,  17,  42, 412, 231, 155, 230])
tensor([0.0705, 0.0173, 0.0398, 0.0074, 0.0027, 0.0067, 0.0653, 0.0366, 0.0246,
        0.0365])<jupyter_text>Let's generate the full grid. There are $\text{maxlen} \times d_\text{model}$ values.__maxlen__ is the maximum position we allow. This has to be predefined.__d_model__ is the size of the input, which is embedding_dim in most cases.<jupyter_code>maxlen = 20
d_model = 12

pe = torch.zeros((maxlen, d_model))
pe.dtype, pe.size()<jupyter_output><empty_output><jupyter_text>__pos__ are the indices of the sequence from 0 to $\text{maxlen}-1$:<jupyter_code>pos = torch.arange(maxlen, dtype=torch.float)
pos<jupyter_output><empty_output><jupyter_text>Reminder:$$\text{PE}_{\text{pos},2i} = \sin(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}), \\\text{PE}_{\text{pos},2i+1} = \cos(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}),$$Let's define the denominator:<jupyter_code>divterm = 10000 ** (torch.arange(0, d_model, step=2) / float(d_model))
divterm
pos.size(), divterm.size()
print((pos[:, None] / divterm).size())
pos[:, None] / divterm
pe[:, ::2] = torch.sin(pos[:, None] / divterm)
sns.heatmap(pe, cmap='RdBu', center=0)
pe[:, 10:]
pe[:, 1::2] = torch.cos(pos[:, None] / divterm)
sns.heatmap(pe, cmap='RdBu', center=0)<jupyter_output><empty_output><jupyter_text>Combining it in a `nn.Module`:<jupyter_code># took inspiration from here: https://pytorch.org/tutorials/beginner/transformer_tutorial.html
class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, maxlen=50):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)
        
        pe = torch.zeros(maxlen, d_model)
        pos = torch.arange(maxlen, dtype=torch.float)
        divterm = 10000 ** (torch.arange(0, d_model, step=2) / float(d_model))
        pe[:, ::2] = torch.sin(pos[:, None] / divterm)
        pe[:, 1::2] = torch.cos(pos[:, None] / divterm)
        
        # Since pe is a constant value not a parameter of the module, we register it as a buffer.
        # Buffers are part of the state dictionary of the module along with parameters.
        # Docs: https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_buffer
        self.register_buffer('pe', pe)

    def forward(self, x):
        # The input sequence may be shorter than maxlen
        seqlen = x.size(0)
        
        # The middle dimension is the batch size.
        # We add it as a dummy dimension.
        x = x + self.pe[:seqlen, None, :]
        return self.dropout(x)
d_model = 12
maxlen = 20
batch_size = 7
seqlen = 10
pos_enc = PositionalEncoding(d_model=d_model, dropout=0., maxlen=maxlen)
x = torch.rand(size=(seqlen, batch_size, d_model))
x_pe = pos_enc(x)
x_pe.size()<jupyter_output><empty_output><jupyter_text>Contextual embeddingsIn GloVe and Word2vec representations, words have static representations, in other words, the same vector is assigned for every occurrence of the word.But words can have different meaning in different contexts, e.g. the word 'stick':1. Find some dry sticks and we'll make a campfire.2. Let's stick with glove embeddings.![elmo](http://jalammar.github.io/images/elmo-embedding-robin-williams.png)_(Peters et. al., 2018 in the ELMo paper)_  ELMo**E**mbeddings from **L**anguage **Mo**delsWord representations are functions of the full sentences instead of the word alone.Two bidirectional LSTM layers are linearly combined.[Deep contextualized word representations](https://arxiv.org/abs/1802.05365) by Peters et al., 2018, 6300 citations  BERT[BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://www.aclweb.org/anthology/N19-1423/)by Devlin et al. 2018, 17500 citations[BERTology](https://huggingface.co/transformers/bertology.html) is the nickname for the growing amount of BERT-related research.Trained on two tasks:1. Masked language model:    1. 15% of the tokenswordpieces are selected at the beginning.    2. 80% of those are replaced with `[MASK]`,    3. 10% are replaced with a random token,    4. 10% are kept intact.    2. Next sentence prediction:    - Are sentences A and B consecutive sentences?    - Generate 50-50%.    - Binary classification task.  Embedding layer<jupyter_code>Image("img/dl/bert_embedding.png")<jupyter_output><empty_output><jupyter_text>Transformer layers Finetuning1. Take a trained BERT model.2. Add a small classification layer on top (typically a 2-layer MLP).3. Train BERT along with the classification layer on an annotated dataset.    - Much smaller than the data BERT was trained onAnother option: freeze BERT and train the classification layer only.- Easier training regime.- Smaller memory footprint.- Worse performance.<jupyter_code>Image("img/dl/bert_encoding_finetuning.png")<jupyter_output><empty_output><jupyter_text>BERT pretrained checkpoints BERT-Base- 12 layers- 12 attention heads per layer- 768 hidden size- 110M parameters BERT-Large- 24 layers- 16 attention heads per layer- 1024 hidden size- 340M parameters Cased and uncasedUncased: everything is lowercased. Diacritics are removed. Multilingual BERT - mBERT104 language version trained on the 100 largest Wikipedia.  BERT implementations[Original Tensorflow implementation](https://github.com/google-research/bert)[Huggingface Transformers](https://huggingface.co/transformers/)- PyTorch implementation originally for BERT-only- Now it supports dozens of other models- Hundreds of other model checkpoints from the community  BERT tokenization WordPiece tokenizerBERT's input **must** be tokenized with BERT's own tokenizer.A middle ground between word and character tokenization.Static vocabulary:- Byte-pair encoding: simple frequency-based tokenization method- Continuation symbols (\\symbol)- Special tokens: `[CLS]`, `[SEP]`, `[MASK]`, `[UNK]`- It tokenizes everything, falling back to characters and `[UNK]` if necessary`AutoTokenizer` is a factory class for pretrained tokenizers. ng id. `from_pretrained` instantiates the corresponding class and loads the weights:<jupyter_code>t = AutoTokenizer.from_pretrained('bert-base-uncased')
print(type(t))
print(len(t.get_vocab()))

t.tokenize("My beagle's name is Tündérke.")
t.tokenize("Русский")<jupyter_output><empty_output><jupyter_text>**Cased** models keep diacritics:<jupyter_code>t = AutoTokenizer.from_pretrained('bert-base-cased')
print(len(t.get_vocab()))

t.tokenize("My beagle's name is Tündérke.")<jupyter_output>28996<jupyter_text>It character tokenizes Chinese and Japanese but doesn't know all the characters:<jupyter_code>t.tokenize("日本語")<jupyter_output><empty_output><jupyter_text>Korean is missing from this version:<jupyter_code>t.tokenize("한 한국어")<jupyter_output><empty_output><jupyter_text>mBERT tokenization104 languages, 1 vocabulary<jupyter_code>t = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
len(t.get_vocab())
t.tokenize("My puppy's name is Tündérke.")
t.tokenize("한 한국어")
t.tokenize("日本語")<jupyter_output><empty_output><jupyter_text>Using BERT Using `BertModel` directly`AutoModel`- each pretrained checkpoint has a string id. `from_pretrained` instantiates the corresponding class and loads the weights:<jupyter_code>tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
model = AutoModel.from_pretrained('bert-base-cased')
type(model), type(tokenizer)
tokenizer.tokenize("There are black cats and black dogs.")<jupyter_output><empty_output><jupyter_text>`__call__` return a dictionary of BERT's encoding:<jupyter_code>tokenizer("There are black cats and black dogs.")<jupyter_output><empty_output><jupyter_text>It can be used for pairs of sentences. Note the values of `token_type_ids`:<jupyter_code>tokenizer("There are black cats and black dogs.", "Another sentence.")<jupyter_output><empty_output><jupyter_text>It can be used for multiple sentences:<jupyter_code>tokenizer(["There are black cats and black dogs.", "There are two white cats."])
encoded = tokenizer(["There are black cats and black dogs.", "There are two white cats."], return_tensors='pt', padding=True)
encoded['attention_mask']<jupyter_output><empty_output><jupyter_text>We need tensors as inputs for BERT:<jupyter_code>encoded = tokenizer("There are black cats and black dogs.", return_tensors='pt')
encoded['input_ids'].size()
encoded
output = model(**encoded, return_dict=True)
output.keys()
output['last_hidden_state'].size(), output['pooler_output'].size()<jupyter_output><empty_output><jupyter_text>Getting all layers:<jupyter_code>output = model(**encoded, output_hidden_states=True, return_dict=True)
output.keys()
len(output['hidden_states']), output['hidden_states'][0].size()<jupyter_output><empty_output><jupyter_text>Remove variable from the global namespace, run the garbage collector:<jupyter_code>del model
gc.collect()<jupyter_output><empty_output><jupyter_text>BERT applications Sequence classificationPretrained model for sentiment analysis.Base model: `distilbert-base-uncased`Finetuned on the [Stanford Sentiment Treebank](https://nlp.stanford.edu/sentiment/index.html) or SST-2, a popular sentiment analysis dataset.Model id: `distilbert-base-uncased-finetuned-sst-2-english`<jupyter_code>nlp = pipeline("sentiment-analysis")
nlp("This is an amazing class.")
nlp("This is not a good class but it's not too bad either.")
nlp("This is not a class.")
del nlp
gc.collect()<jupyter_output><empty_output><jupyter_text>Sequence tagging/labeling: Named entity recognitionBase model: `bert-large-cased`Finetuned on [CoNLL-2003 NER](https://www.clips.uantwerpen.be/conll2003/ner/).<jupyter_code>nlp = pipeline("ner")
result = nlp("Jupiter is a planet that orbits around James the center of the Universe")
result
result = nlp("<NAME> has a pet pig named Estella.")
result
del nlp
gc.collect()<jupyter_output><empty_output><jupyter_text>Machine translation<jupyter_code>nlp = pipeline("translation_en_to_fr")
print(nlp("Hugging Face is a technology company based in New York and Paris", max_length=40))<jupyter_output>Some weights of T5Model were not initialized from the model checkpoint at t5-base and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.<jupyter_text>Even the [blessé - blessed false cognate](https://frenchtogether.com/french-english-false-friends/) is handled correctly:<jupyter_code>nlp("I was blessed by God after I injured my head.", max_length=40)
gc.collect()
del nlp
gc.collect()<jupyter_output><empty_output><jupyter_text>Masked language modelingUses `distilroberta-base`<jupyter_code>nlp = pipeline("fill-mask")
prompt = "Twitter is a bad idea /s> [MASK]"

for n in range(10):
    result = nlp(f"{prompt} {nlp.tokenizer.mask_token}")
    token = result[0]['token_str'][1:]
    prompt += " " + token
    
prompt
from pprint import pprint
pred = nlp(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks.")
pprint(pred)
pred = nlp(f"{nlp.tokenizer.mask_token} is a very good idea.")
pprint(pred)
pred = nlp(f"{nlp.tokenizer.mask_token} is a bad idea.")
pprint(pred)
del nlp
gc.collect()<jupyter_output><empty_output><jupyter_text>Other models Pretrained modelsRoBERTa: identical model, larger training data, different training objectiveDistilBERT: smaller version of BERT. It was _distilled_ or compressed from BERT with a student-teacher setup.ALBERT: smaller BERTXLM-RoBERTa: multilingual version of RoBERTaDistil-mBERT: distilled multilingual BERT  Community models[Over 1000 community contributions](https://huggingface.co/models) huBERTThe first Hungarian-only model and the only one registered on Huggingface.Other models are available at https://hilanco.github.io/.BERT base, trained on Webcorpus 2.0, a version of CommonCrawl.Its tokenizer works much better for Hungarian than mBERT's:<jupyter_code>hubert_tokenizer = AutoTokenizer.from_pretrained('SZTAKI-HLT/hubert-base-cc')
# hubert = AutoModel.from_pretrained('SZTAKI-HLT/hubert-base-cc')
sent = ("<NAME>yarországról szóló, az Orbán-kormányt kritizáló levelére miniszteri és "
        "államtitkári szinten is reagált a magyar kormány.")
hubert_tokenizer.tokenize(sent)
bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
bert_tokenizer.tokenize(sent)<jupyter_output><empty_output><jupyter_text>GPT-2 text generationCausal language modeliing is when the $i^{th}$ token is modeled based on all the previous tokens as opposed to masked language modeling where both left and right context are used.<jupyter_code>text_generator = pipeline("text-generation")
print(text_generator("This is a serious issue we should address", max_length=50, do_sample=False)[0]['generated_text'])
print(text_generator("Twitter is a bad idea, <NAME> had a bad day when he came up with it", max_length=100, do_sample=False)[0]['generated_text'])
del text_generator
gc.collect()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup