<jupyter_start><jupyter_text>clone repository and move `persian_re` package to use it<jupyter_code>!git clone https://github.com/nimaafshar/persian_relation_extraction.git
!mv persian_relation_extraction/persian_re/ persian_re/<jupyter_output><empty_output><jupyter_text>installing specified requirements for google colab<jupyter_code>!pip install -r persian_relation_extraction/requirements_colab.txt<jupyter_output>Collecting transformers==4.17.0
  Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)
[K     |████████████████████████████████| 3.8 MB 5.4 MB/s 
[?25hCollecting hazm==0.7.0
  Downloading hazm-0.7.0-py3-none-any.whl (316 kB)
[K     |████████████████████████████████| 316 kB 47.2 MB/s 
[?25hCollecting pyyaml==5.4.1
  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
[K     |████████████████████████████████| 636 kB 47.3 MB/s 
[?25hCollecting clean-text[gpl]
  Downloading clean_text-0.6.0-py3-none-any.whl (11 kB)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->-r persian_relation_extraction/requirements_colab.txt (line 1)) (21.3)
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.17.0->-r persian_relation_extraction/requirements_colab.txt (line 1)) (3.6.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from[...]<jupyter_text>checking *CUDA* availablity<jupyter_code>!nvidia-smi<jupyter_output>Mon Mar 21 20:10:06 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   64C    P8    32W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-------[...]<jupyter_text>Entity-Marker tokenizer + Entity Start output model Imports<jupyter_code>import numpy as np

import torch, gc
import torch.nn as nn
from torch.optim import AdamW

from transformers import BertConfig
from transformers import get_linear_schedule_with_warmup

import collections

from sklearn.metrics import classification_report
from persian_re.preprocess import PerlexData, create_data_loader
from persian_re.tokenizers import BertEntityMarkerTokenizer
from persian_re.settings import Config
from persian_re.models import EntityStartModel
from persian_re.operation import Trainer,TrainingArguments<jupyter_output><empty_output><jupyter_text>GPU configuration<jupyter_code>device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(f'device: {device}')

train_on_gpu = torch.cuda.is_available()

if not train_on_gpu:
    print('CUDA is not available.  Training on CPU ...')
else:
    print('CUDA is available!  Training on GPU ...')
#clear cuda cache
gc.collect()
torch.cuda.empty_cache()<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>data = PerlexData.get_instance()<jupyter_output><empty_output><jupyter_text>Entity Marker tokenizer<jupyter_code>tokenizer: BertEntityMarkerTokenizer = BertEntityMarkerTokenizer.from_pretrained(Config.MODEL_NAME_OR_PATH)<jupyter_output><empty_output><jupyter_text>Configuration<jupyter_code>arguments = TrainingArguments(epochs=20,
                              clip=0.0,
                              train_callback_interval=100)
config = BertConfig.from_pretrained(
    Config.MODEL_NAME_OR_PATH, **{
        'label2id': data.label2ids,
        'id2label': data.id2labels,
        'hidden_dropout_prob': 0.2
    })
config<jupyter_output><empty_output><jupyter_text>*DataLoader* objects<jupyter_code>train_data_loader = create_data_loader(data.x_train, data.y_train, tokenizer, Config.MAX_LEN,
                                       Config.TRAIN_BATCH_SIZE, data.label2ids)
valid_data_loader = create_data_loader(data.x_valid, data.y_valid, tokenizer, Config.MAX_LEN,
                                       Config.VALID_BATCH_SIZE, data.label2ids)
test_data_loader = create_data_loader(data.x_test,data.y_test, tokenizer, Config.MAX_LEN, Config.TEST_BATCH_SIZE, data.label2ids)<jupyter_output><empty_output><jupyter_text>[CLS] Relation Extraction Model<jupyter_code>pt_model = EntityStartModel(config,tokenizer.e1_start_token_id,tokenizer.e2_start_token_id)
pt_model.resize_token_embeddings(len(tokenizer))
pt_model = pt_model.to(device)<jupyter_output><empty_output><jupyter_text>Trainingwith:- **AdamW** optimizer with initial learning rate `INITIAL_LEARNING_RATE`- **Linear Scheduler** with no warmup- **CrossEntropyLoss** with class weights to balance data<jupyter_code>optimizer = AdamW(pt_model.parameters(), lr=Config.INITIAL_LEARNING_RATE)
total_steps = len(train_data_loader) * arguments.epochs
scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=0,
    num_training_steps=total_steps
)

loss_fn = nn.CrossEntropyLoss(weight=torch.FloatTensor(data.class_weights).to(device))
trainer = Trainer(pt_model,device,loss_fn,optimizer,scheduler,arguments,train_data_loader,valid_data_loader,test_data_loader)<jupyter_output><empty_output><jupyter_text>create dir `output` in `BASE_PATH` to save model<jupyter_code>!mkdir persian_relation_extraction/output/
trainer.train()<jupyter_output><empty_output><jupyter_text>Model Evaluation and Comparison1. using **last epoch** (overfitted) model<jupyter_code>y_pred, y_pred_probs, y_true, metrics = trainer.predict_tests()
metrics
print(classification_report(y_true, y_pred,target_names=data.labels))<jupyter_output>precision    recall  f1-score   support

      Cause-Effect(e1,e2)       0.88      0.92      0.90        99
      Cause-Effect(e2,e1)       0.83      0.84      0.84       143
   Component-Whole(e1,e2)       0.76      0.75      0.75       119
   Component-Whole(e2,e1)       0.83      0.65      0.72       110
 Content-Container(e1,e2)       0.80      0.88      0.84       113
 Content-Container(e2,e1)       0.84      0.90      0.87        29
Entity-Destination(e1,e2)       0.85      0.87      0.86       214
     Entity-Origin(e1,e2)       0.77      0.77      0.77       155
     Entity-Origin(e2,e1)       0.94      0.83      0.88        35
 Instrument-Agency(e1,e2)       0.59      0.62      0.61        16
 Instrument-Agency(e2,e1)       0.76      0.72      0.74        99
 Member-Collection(e1,e2)       0.60      0.75      0.67        24
 Member-Collection(e2,e1)       0.79      0.75      0.77       148
     Message-Topic(e1,e2)       0.78      0.88      0.83     [...]<jupyter_text>2. using **saved model** (which has the lowest validation loss)<jupyter_code>saved_model = torch.load(Config.OUTPUT_PATH)
temp_trainer = Trainer(saved_model,device,loss_fn,optimizer,scheduler,arguments,train_data_loader,valid_data_loader,test_data_loader)
y_pred, y_pred_probs, y_true, metrics = temp_trainer.predict_tests()
metrics
print(classification_report(y_true, y_pred,target_names=data.labels))<jupyter_output>precision    recall  f1-score   support

      Cause-Effect(e1,e2)       0.81      0.95      0.87        99
      Cause-Effect(e2,e1)       0.80      0.90      0.84       143
   Component-Whole(e1,e2)       0.64      0.81      0.72       119
   Component-Whole(e2,e1)       0.72      0.56      0.63       110
 Content-Container(e1,e2)       0.80      0.84      0.82       113
 Content-Container(e2,e1)       0.79      0.93      0.86        29
Entity-Destination(e1,e2)       0.93      0.75      0.83       214
     Entity-Origin(e1,e2)       0.70      0.85      0.77       155
     Entity-Origin(e2,e1)       0.63      0.83      0.72        35
 Instrument-Agency(e1,e2)       0.38      0.75      0.50        16
 Instrument-Agency(e2,e1)       0.76      0.68      0.72        99
 Member-Collection(e1,e2)       0.44      0.83      0.58        24
 Member-Collection(e2,e1)       0.84      0.71      0.77       148
     Message-Topic(e1,e2)       0.90      0.69      0.78     [...]<jupyter_text>Saving Best Model<jupyter_code>torch.save(pt_model,'/content/drive/MyDrive/RE_task/entitymarker_entitystart_v0.1.bin')<jupyter_output><empty_output><jupyter_text>How to prevent overfitting1. train only last n layers of BERT2. increase dropout probablity3. change scheduler (use `ReduceOnPlateu` or non-linear schedulers)<jupyter_code><jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup