# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/xbox10/ru-gpts/blob/master/Finetune_Twitter_GPT3Small.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="3RHDK81QqrET"
# # Finetune ruGPT3Small on your data
#

# + [markdown] id="eK10D3MSpYty"
# ## Install enviroment

# + id="asqMueYPeIgK"
# !pip3 install urllib3==1.25.4

# + id="yPqtVgbkeTx7"
# !pip3 install transformers==2.8.0

# + id="mpkjTWefecLc"
# !wget https://raw.githubusercontent.com/sberbank-ai/ru-gpts/master/pretrain_transformers.py

# + id="L7zu3BqpqJQ7"
# !wget https://raw.githubusercontent.com/sberbank-ai/ru-gpts/master/generate_transformers.py

# + id="GtAeHbhbTnzO"
# %%writefile setup.sh

export CUDA_HOME=/usr/local/cuda-10.1
git clone https://github.com/NVIDIA/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex

# + id="udZ7AiMWTpD9"
# !sh setup.sh

# + [markdown] id="DP7YAlDPqknI"
# ## Add data to colab
#

# + id="HddOX2tmKOoW"
# !wget https://raw.githubusercontent.com/FurkanArslan/twitter-text-classification/master/twitter_sentiment_corpus.csv

# + id="f5EtK-jerBRv"
data_path = "twitter_sentiment_corpus.csv"

# + [markdown] id="GDYi1TVTrtkO"
# ## Prepare data

# + id="OXdNbrq3rgzq"
import pandas as pd
data = pd.read_csv(data_path)
data.head()

# + id="Izc0lkkHr2Rz"
import numpy as np
import random

# + id="3tkc1_49r36X"
random.seed(1234)
np.random.seed(1234)

# + id="-IWaigRvyxmT"
data.shape

# + id="WsC7iXoNy1eY"
val_ind = random.sample(range(data.shape[0]), 500)

# + id="j0nbtCxLzTfM"
train = [data.iloc[i]['TweetText'] for i in range(len(data)) if i not in val_ind][:10000]
valid = [data.iloc[i]['TweetText'] for i in range(len(data)) if i in val_ind]

# + id="2T0gN6gqr9pa"
len(train), len(valid)

# + id="ZPB8rrVPr-kh"
with open("train.txt", "w") as file:
    file.write("\n".join(train))

# + id="HP5_nk_0sAB0"
with open("valid.txt", "w") as file:
    file.write("\n".join(valid))

# + [markdown] id="NitGcEKPsDQE"
# ## Run finetuning
# The following code download our model and tokenizer from transformers and finetune model essays.
#
# This took aroung ten minutes and obtain perplexity = tensor(13.8065)

# + id="5vL07XFvsBBU"
# !python pretrain_transformers.py \
#     --output_dir=comment_model \
#     --model_type=gpt2 \
#     --model_name_or_path=sberbank-ai/rugpt3medium_based_on_gpt2 \
#     --do_train \
#     --train_data_file=train.txt \
#     --do_eval \
#     --fp16 \
#     --eval_data_file=valid.txt \
#     --per_gpu_train_batch_size 1 \
#     --gradient_accumulation_steps 1 \
#     --num_train_epochs 5 \
#     --block_size 2048 \
#     --overwrite_output_dir

# + [markdown] id="w2N6ylGPt1F5"
# ## Check our model

# + id="JRlAAsIbsHdf"
# !python generate_transformers.py \
#     --model_type=gpt2 \
#     --model_name_or_path=gpt2small \
#     --k=6 \
#     --p=0.94 \
#     --length=100

# + id="-2q_3DmPsgLN"


# + [markdown] id="Vzn-sWlp6vYt"
# ## Some examples I got
#

# + [markdown] id="G6FrdJGfoOCo"
# Context >>> Is it
#
# ruGPT:
# Is it's the case that #Microsoft is using #SAP to manage #Microsoft’s business?

# + [markdown] id="9xrK05vCoUXh"
# Context >>> It  is
#
# ruGPT:
# It is the only one that I've been trying to do.
# Siri is on the verge of death, and I'm not aware of why she has to be on the verge.

# + [markdown] id="aOo1x1ArogLs"
# Context >>> Why
#
# ruGPT:
# Why do I go here?  I'm wondering if I can use the #Microsoft Windows 8 Pro for the #WindowsPhone phone?  Thanks! 

# + [markdown] id="D34DPrwvozgd"
#

StarcoderdataJupyterScriptsDedupFiltered