<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  GenSen with PytorchIn this tutorial, you will train a GenSen model for the sentence similarity task. We use the [SNLI](https://nlp.stanford.edu/projects/snli/) dataset in this example. For a more detailed walkthrough about data processing jump to [SNLI Data Prep](../01-prep-data/snli.ipynb). A quickstart version of this notebook can be found [here](../00-quick-start/) Notes:The model training part of this notebook can only run on a GPU machine. The running time shown in the notebook is on a Standard_NC6 Azure VM with 1 NVIDIA Tesla K80 GPU and 12 GB GPU memory. See the [README](README.md) for more details of the running time.  Overview What is GenSen?GenSen is a technique to learn general purpose, fixed-length representations of sentences via multi-task training. GenSen model combines the benefits of diverse sentence-representation learning objectives into a single multi-task framework. "This is the first large-scale reusable sentence representation model obtained by combining a set of training objectives with the level of diversity explored here, i.e. multi-lingual NMT, natural language inference, constituency parsing and skip-thought vectors." [\[1\]](References) These representations are useful for transfer and low-resource learning. GenSen is trained on several data sources with multiple training objectives on over 100 milion sentences.The GenSen model is most similar to that of Luong et al. (2015) [\[4\]](References), who train a many-to-many **sequence-to-sequence** model on a diverse set of weakly related tasks that includes machine translation, constituency parsing, image captioning, sequence autoencoding, and intra-sentence skip-thoughts. However, there are two key differences. "First, like McCann et al. (2017) [\[5\]](References), their use of an attention mechanism prevents learning a ﬁxed-length vector representation for a sentence. Second, their work aims for improvements on the same tasks on which the model is trained, as opposed to learning re-usable sentence representations that transfer elsewhere." [\[1\]](References) Why GenSen?GenSen model performs the state-of-the-art results on multiple datasets, such as MRPC, SICK-R, SICK-E and STS, for sentence similarity. The reported results are as follows compared with other models [\[3\]](References):| Model | MRPC | SICK-R | SICK-E | STS || --- | --- | --- | --- | --- || GenSen (Subramanian et al., 2018) | 78.6/84.4 | 0.888 | 87.8 | 78.9/78.6 || [InferSent](https://arxiv.org/abs/1705.02364) (Conneau et al., 2017) | 76.2/83.1 | 0.884 | 86.3 | 75.8/75.5 || [TF-KLD](https://www.aclweb.org/anthology/D13-1090) (Ji and Eisenstein, 2013) | 80.4/85.9 | - | - | - |  OutlineThis notebook is organized as follows:1. Data preparation and inspection.2. Model training and prediction.For a more detailed deep dive of the Gensen model checkout the [Gensen Deep Dive Notebook](gensen_aml_deep_dive.ipynb)  0. Global Settings<jupyter_code>import sys
sys.path.append("../..")

import os
import papermill as pm
import scrapbook as sb

from utils_nlp.dataset.preprocess import to_lowercase, to_nltk_tokens
from utils_nlp.dataset import snli, preprocess
from utils_nlp.models.pretrained_embeddings.glove import download_and_extract
from utils_nlp.dataset import Split
from examples.sentence_similarity.gensen_wrapper import GenSenClassifier

print("System version: {}".format(sys.version))
max_epoch = None
config_filepath = 'gensen_config.json'
base_data_path = '../../data'
nrows = None<jupyter_output><empty_output><jupyter_text>1. Data Preparation and inspection The [SNLI](https://nlp.stanford.edu/projects/snli/) corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).   1.1 Load the datasetWe provide a function load_pandas_df which does the following* Downloads the SNLI zipfile at the specified directory location* Extracts the file based on the specified split* Loads the split as a pandas dataframe The zipfile contains the following files:    * snli_1.0_dev.txt    * snli_1.0_train.txt    * snli_1.0_test.tx    * snli_1.0_dev.jsonl    * snli_1.0_train.jsonl    * snli_1.0_test.jsonl    The loader defaults to reading from the .txt file; however, you can change this to .jsonl by setting the optional file_type parameter when calling the function.<jupyter_code>train = snli.load_pandas_df(base_data_path, file_split=Split.TRAIN, nrows=nrows)
dev = snli.load_pandas_df(base_data_path, file_split=Split.DEV, nrows=nrows)
test = snli.load_pandas_df(base_data_path, file_split=Split.TEST, nrows=nrows)

train.head()<jupyter_output><empty_output><jupyter_text>1.2 TokenizeWe have loaded the dataset into pandas.DataFrame, we now convert sentences to tokens. We also clean the data before tokenizing. This includes dropping unneccessary columns and renaming the relevant columns as score, sentence_1, and sentence_2.<jupyter_code>def clean_and_tokenize(df):
    df = snli.clean_cols(df)
    df = snli.clean_rows(df)
    df = preprocess.to_lowercase(df)
    df = preprocess.to_nltk_tokens(df)
    return df

train = clean_and_tokenize(train)
dev = clean_and_tokenize(dev)
test = clean_and_tokenize(test)<jupyter_output><empty_output><jupyter_text>Once we have the clean pandas dataframes, we do lowercase standardization and tokenization. We use the [NLTK] (https://www.nltk.org/) library for tokenization.<jupyter_code>dev.head()<jupyter_output><empty_output><jupyter_text>2. Model application, performance and analysis of the resultsThe model has been implemented as a GenSen class with the specifics hidden inside the fit() method, so that no explicit call is needed. The algorithm operates in three different steps:** Model initialization ** : This is where we tell our class how to train the model. The main parameters to specify are the number of1. config file which contains information about the number of training epochs, the minibatch size etc.2. cache_dir which is the folder where all the data will be saved.3. learning rate for the model4. path to the pretrained embedding vectors.** Model fit ** : This is where we train the model on the data. The method takes two arguments: the training, dev and test set pandas dataframes. Note that the model is trained only on the training set, the test set is used to display the test set accuracy of the trained model, that in turn is an estimation of the generazation capabilities of the algorithm. It is generally useful to look at these quantities to have a first idea of the optimization behaviour.** Model prediction ** : This is where we generate the similarity for a pair of sentences. Once the model has been trained and we are satisfied with its overall accuracy we use the saved model to show the similarity between two provided sentences.   2.0 Download pretrained vectorsIn this example we use gloVe for pretrained embedding vectors.<jupyter_code>pretrained_embedding_path = download_and_extract(base_data_path)<jupyter_output>Vector file already exists. No changes made.<jupyter_text>2.1 Initialize Model<jupyter_code>clf = GenSenClassifier(config_file = config_filepath, 
                       pretrained_embedding_path = pretrained_embedding_path,
                       learning_rate = 0.0001, 
                       cache_dir=base_data_path,
                      max_epoch=max_epoch)<jupyter_output><empty_output><jupyter_text>2.2 Train Model<jupyter_code>%%time
clf.fit(train, dev, test)<jupyter_output>/data/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/modules/rnn.py:46: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.8 and num_layers=1
  "num_layers={}".format(dropout, num_layers))
../../examples/sentence_similarity/gensen_train.py:431: UserWarning: torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.
  torch.nn.utils.clip_grad_norm(model.parameters(), 1.0)
../../utils_nlp/models/gensen/utils.py:364: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  Variable(torch.LongTensor(sorted_src_lens), volatile=True)
/data/anaconda/envs/nlp_gpu/lib/python3.6/site-packages/torch/nn/functional.py:1332: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
  warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
/data/anaconda/envs/nlp_gpu/lib/python[...]<jupyter_text>2.3 PredictIn the predict method we perform Pearson's Correlation computation [\[2\]](References) on the outputs of the model. The predictions of the model can be further improved by hyperparameter tuning which we walk through in the other example [here](gensen_aml_deep_dive.ipynb).<jupyter_code>sentences = [
        'The sky is blue and beautiful',
        'Love this blue and beautiful sky!'
    ]

results = clf.predict(sentences)
print("******** Similarity Score for sentences **************")
print(results)

# Record results with scrapbook for tests
sb.glue("results", results.to_dict())<jupyter_output>******** Similarity Score for sentences **************
          0         1
0  1.000000  0.966793
1  0.966793  1.000000