# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] colab_type="text" id="aXpIRSAQ-cw1"
# # Getting started with DeepMatcher
#
# Note: you can run **[this notebook live in Google Colab](https://colab.research.google.com/github/anhaidgroup/deepmatcher/blob/master/examples/getting_started.ipynb)** and use free GPUs provided by Google.
#
# This tutorial describes how to effortlessly perform entity matching using deep neural networks. Specifically, we will see how to match pairs of tuples (also called data records or table rows) to determine if they refer to the same real world entity. To do so, we will need labeled examples as input, i.e., tuple pairs which have been annotated as matches or non-matches. This will be used to train our neural network using supervised learning. At the end of this tutorial, you will have a trained neural network as output which you can easily apply to unlabeled tuple pairs to make predictions.

# + [markdown] colab_type="text" id="19SozaN8QJOf"
# As an overview, here are the 4 steps to use `deepmatcher` which we will go through in this tutorial:
#
# <ol start="0">
#   <li>Setup</li>
#   <li>Process labeled data</li>
#   <li>Define neural network model</li>
#   <li>Train model</li>
#   <li>Apply model to new data</li>
# </ol>
#
# Let's begin!

# + [markdown] colab_type="text" id="sBtZVvrSQJOf"
# ## Step 0. Setup
#
# If you are running this notebook inside Colab, you will first need to install necessary packages by running the code below:

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 17} colab_type="code" executionInfo={"elapsed": 82136, "status": "ok", "timestamp": 1524617064270, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="LEiDiDNz-PEG" outputId="3af884c1-b09b-4659-89da-eb1c24c811c7"
try:
    import deepmatcher
except:
    # !pip install -qqq deepmatcher

# + [markdown] colab_type="text" id="D1Hmd9LXQJOk"
# Now let's import `deepmatcher` which will do all the heavy lifting to build and train neural network models for entity matching. 

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 17} colab_type="code" executionInfo={"elapsed": 6623, "status": "ok", "timestamp": 1524617080672, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="0422OsKBQJOk" outputId="1f39e87e-380f-4ce2-95bd-bea04867553e"
import deepmatcher as dm

# + [markdown] colab_type="text" id="g6inOGkQQJOn"
# We recommend having a GPU available for the training in Step 4. In case a GPU is not available, we will use all available CPU cores. You can run the following command to determine if a GPU is available and will be used for training:

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 34} colab_type="code" executionInfo={"elapsed": 289, "status": "ok", "timestamp": 1524617081918, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="GYb5r0etQJOo" outputId="ec451859-b108-4b36-effe-48b62ccb04a4"
import torch
torch.cuda.is_available()

# + [markdown] colab_type="text" id="Dv_sFh1_QJOs"
# ### Download sample data for entity matching
#
# Now let's get some sample data to play with in this tutorial. We will need three sets of labeled data and one set of unlabeled data:
#
# 1. **Training Data:** This is used for training our neural network model.
# 2. **Validation Data:** This is used for determining the configuration (i.e., hyperparameters) of our model in such a way that the model does not overfit to the training set.
# 3. **Test Data:** This is used to estimate the performance of our trained model on unlabeled data.
# 4. **Unlabeled Data:** The trained model is applied on this data to obtain predictions, which can then be used for downstream tasks in practical application scenarios.
#
# We download these four data sets to the `sample_data` directory:

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 17} colab_type="code" executionInfo={"elapsed": 6944, "status": "ok", "timestamp": 1524617092992, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="gvPSNrnUuHNF" outputId="cc8da365-7936-4bdd-9ee9-9e2dc572289d"
# !mkdir -p sample_data/itunes-amazon
# !wget -qnc -P sample_data/itunes-amazon https://raw.githubusercontent.com/anhaidgroup/deepmatcher/master/examples/sample_data/itunes-amazon/train.csv
# !wget -qnc -P sample_data/itunes-amazon https://raw.githubusercontent.com/anhaidgroup/deepmatcher/master/examples/sample_data/itunes-amazon/validation.csv
# !wget -qnc -P sample_data/itunes-amazon https://raw.githubusercontent.com/anhaidgroup/deepmatcher/master/examples/sample_data/itunes-amazon/test.csv
# !wget -qnc -P sample_data/itunes-amazon https://raw.githubusercontent.com/anhaidgroup/deepmatcher/master/examples/sample_data/itunes-amazon/unlabeled.csv

# + [markdown] colab_type="text" id="lu6QgmbDQJOv"
# To get an idea of how our data looks like, let's take a peek at the training dataset:

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 425} colab_type="code" executionInfo={"elapsed": 314, "status": "ok", "timestamp": 1524617124996, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="lpsH6NEZQJOw" outputId="358b276f-f91a-412b-ea2f-54fc567b8a36"
import pandas as pd
pd.read_csv('sample_data/itunes-amazon/train.csv').head()

# + [markdown] colab_type="text" id="KuCrlwjeQs8i"
# ## Step 1. Process labeled data
#
# Before we can use our data for training, `deepmatcher` needs to first load and process it in order to prepare it for neural network training. Currently `deepmatcher` only supports processing CSV files. Each CSV file is assumed to have the following kinds of columns:
#
# * **"Left" attributes (required):** Our goal is to match tuple pairs. "Left" attributes are columns that correspond to the "left" tuple or the first tuple in the tuple pair. These column names are expected to be prefixed with "left_" by default.
# * **"Right" attributes (required):** "Right" attributes are columns that correspond to the "right" tuple or the second tuple in the tuple pair. These column names are expected to be prefixed with "right_" by default.
# * **Label column (required for train, validation, test):** Column containing the labels (match or non-match) for each tuple pair. Expected to be named "label" by default
# * **ID column (required):** Column containing a unique ID for each tuple pair. This is for evaluation convenience.  Expected to be named "id" by default.
#
# More details on what data processing involves and ways to customize it are described in **[this notebook](https://nbviewer.jupyter.org/github/anhaidgroup/deepmatcher/blob/master/examples/data_processing.ipynb)**. 
#
# ### Processing labeled data
# In order to process our train, validation and test CSV files we call `dm.data.process` in the following code snippet which will load and process the CSV files and return three processed `MatchingDataset` objects respectively. These dataset objects will later be used for training and evaluation. The basic parameters to `dm.data.process` are as follows:
#
# * **path (required): ** The path where all data is stored. This includes train, validation and test. `deepmatcher` may create new files in this directory to store information about these data sets. This allows subsequent `dm.data.process` calls to be much faster.
# * **train (required): ** File name of training data in `path` directory.
# * **validation (required): ** File name of validation data in `path` directory.
# * **test (optional): ** File name of test data in `path` directory.
# * **ignore_columns (optional): ** Any columns in the CSV files that you may want to ignore for the purposes of training. These should be included here. 
#
# Note that the train, validation and test CSVs must all share the same schema, i.e., they should have the same columns. Processing data involves several steps and can take several minutes to complete, especially if this is the first time you are running the `deepmatcher` package.
#
# NOTE: If you are running this in Colab, you may get a message saying 'Memory usage is close to the limit.' You can safely ignore it for now. We are working on reducing the memory footprint.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 227} colab_type="code" executionInfo={"elapsed": 120549, "status": "ok", "timestamp": 1524617249949, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="WJ37wi1HC9EJ" outputId="6c19622b-1d60-4e01-e810-6ad99173040c"
train, validation, test = dm.data.process(
    path='sample_data/itunes-amazon',
    train='train.csv',
    validation='validation.csv',
    test='test.csv')

# + [markdown] colab_type="text" id="4RCEIm8MQJO3"
# #### Peeking at processed data
# Let's take a look at how the processed data looks like. To do this, we get the raw `pandas` table corresponding to the processed training dataset object. 

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 460} colab_type="code" executionInfo={"elapsed": 333, "status": "ok", "timestamp": 1524617336979, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="qmkzAzWOQJO3" outputId="35aed066-c9d9-406b-e6cc-ce7dcd27029a"
train_table = train.get_raw_table()
train_table.head()

# + [markdown] colab_type="text" id="hpo4JpshQJO6"
# The processed attribute values have been tokenized and lowercased so they may not look exactly the same as the input training data. These modifications help the neural network generalize better, i.e., perform better on data not trained on. 

# + [markdown] colab_type="text" id="8mr3oVzuQJO6"
# ## Step 2. Define neural network model
#
# In this step you tell `deepmatcher` what kind of neural network you would like to use for entity matching. The easiest way to do this is to use one of the several kinds of neural network models that comes built-in with `deepmatcher`. To use a built-in network, construct a `dm.MatchingModel` as follows:
#
# `model = dm.MatchingModel(attr_summarizer='<TYPE>')`
#
# where `<TYPE>` is one of `sif`, `rnn`, `attention` or `hybrid`. If you are not familiar with what these mean, we strongly recommend taking a look at either **[slides from our talk on deepmatcher](http://bit.do/deepmatcher-talk)** for a high level overview, or **[our paper](http://pages.cs.wisc.edu/~anhai/papers1/deepmatcher-sigmod18.pdf)** for a more detailed explanation. Here we give briefly describe the intuition behind these four model types:
# * **sif:** This model considers the **words** present in each attribute value pair to determine a match or non-match. It does not take word order into account.
# * **rnn:** This model considers the **sequences of words** present in each attribute value pair to determine a match or non-match.
# * **attention:** This model considers the **alignment of words** present in each attribute value pair to determine a match or non-match. It does not take word order into account.
# * **hybrid:** This model considers the **alignment of sequences of words** present in each attribute value pair to determine a match or non-match. This is the default.
#
# `deepmatcher` is highly customizable and allows you to tune almost every aspect of the neural network model for your application scenario. **[This tutorial](https://nbviewer.jupyter.org/github/anhaidgroup/deepmatcher/blob/master/examples/matching_models.ipynb)** discusses the structure of `MatchingModel`s and how they can be customized.
#
# For this tutorial, let's create a `hybrid` model for entity matching:

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 17} colab_type="code" executionInfo={"elapsed": 234, "status": "ok", "timestamp": 1524617340683, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="NwUE2jMXQJO7" outputId="e05a0482-944b-4a68-a1c4-32e40ba3f17a"
model = dm.MatchingModel(attr_summarizer='hybrid')

# + [markdown] colab_type="text" id="7iLAHpuVQJO9"
# ## Step 3. Train model
#
# Next, we train the defined neural network model using the processed training and validation data. To do so, we call the `run_train` method which takes the following basic parameters:
#
# * **train:** The processed training dataset object (of type `MatchingDataset`).
# * **validation:** The processed validation dataset object (of type `MatchingDataset`).
# * **epochs:** Number of times to go over the entire `train` data for training the model.
# * **batch_size:** Number of labeled examples (tuple pairs) to use for each training step. This value may be increased if you have a lot of training data and would like to speed up training. The optimal value is dataset dependent.
# * **best_save_path:** Path to save the best model.
# * **pos_neg_ratio**: The ratio of the weight of positive examples (matches) to weight of negative examples (non-matches). This value should be increased if you have fewer matches than non-matches in your data. The optimal value is dataset dependent.
#
# Many other aspects of the training algorithm can be customized. For details on this, please refer the API documentation for **[run_train]()**

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 2815} colab_type="code" executionInfo={"elapsed": 297777, "status": "ok", "timestamp": 1524617643666, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="RIZclsK1QJO-" outputId="f2ce7d13-0bd4-454e-aafc-d19af02e92b0"
model.run_train(
    train,
    validation,
    epochs=10,
    batch_size=16,
    best_save_path='hybrid_model.pth',
    pos_neg_ratio=3)

# + [markdown] colab_type="text" id="lejVqpjAQJPD"
# ## Step 4. Apply model to new data
#
# ### Evaluating on test data
# Now that we have a trained model for entity matching, we can now evaluate its accuracy on test data, to estimate the performance of the model on unlabeled data.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 86} colab_type="code" executionInfo={"elapsed": 2111, "status": "ok", "timestamp": 1524617885197, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="jLoneAI_QJPD" outputId="2d8a56e0-8ab9-4826-a09f-ffa838eb4eed"
# Compute F1 on test set
model.run_eval(test)

# + [markdown] colab_type="text" id="W4ixyezcQJPG"
# ### Evaluating on unlabeled data
#
# We finally apply the trained model to unlabeled data to get predictions. To do this, we need to first process the unlabeled data.
#
# #### Processing unlabeled data
#
# To process unlabeled data, we use `dm.data.process_unlabeled`, as shown in the code snippet below. The basic parameters for this call are as follows:
#
# * **path (required): ** The full path to the unlabeled data file (not just the directory).
# * **trained_model (required): ** The trained model. The model is aware of the configuration of the training data on which it was trained, and so `deepmatcher` reuses the same configuration for the unlabeled data.
# * **ignore_columns (optional): ** Any columns in the unlabeled CSV file that you may want to ignore for the purposes of evaluation. If not specified, the columns that were ignored while processing the training set will also be ignored while processing the unlabeled data.
#
# Note that the unlabeled CSV file must have the same schema as the train, validation and test CSVs.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 52} colab_type="code" executionInfo={"elapsed": 725, "status": "ok", "timestamp": 1524617889802, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="ZmZ7EAtmQJPH" outputId="dfbe84b0-996b-49f0-caf0-b3961e9b2c70"
unlabeled = dm.data.process_unlabeled(
    path='sample_data/itunes-amazon/unlabeled.csv',
    trained_model=model)

# + [markdown] colab_type="text" id="OF18CNWbQJPM"
# #### Obtaining predictions
#
# Next, we call the `run_prediction` method which takes a processed data set object and returns a `pandas` dataframe containing tuple pair IDs (`id` column) and the corresponding match score predictions (`match_score` column). `match_scores` are in [0, 1] and a score above 0.5 indicates a match prediction.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 314} colab_type="code" executionInfo={"elapsed": 4427, "status": "ok", "timestamp": 1524617897938, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="0UE0Lg4KQJPN" outputId="3a518e46-7b56-4cf7-9df1-ba440ea9becf"
predictions = model.run_prediction(unlabeled)
predictions.head()

# + [markdown] colab_type="text" id="vXclEMhkQJPR"
# You may optionally set the `output_attributes` parameter to also include all attributes present in the original input table. As mentioned earlier, the processed attribute values will likely look a bit different from the attribute values in the input CSV files due to modifications such as tokenization and lowercasing.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 680} colab_type="code" executionInfo={"elapsed": 4414, "status": "ok", "timestamp": 1524617906156, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="0wPnH3NPQJPS" outputId="f0976d43-3c45-49e6-ac10-78a40eed81e0"
predictions = model.run_prediction(unlabeled, output_attributes=True)
predictions.head()

# + [markdown] colab_type="text" id="bJxMcI0eQJPU"
# You can then save these predictions to CSV and use them for downstream tasks.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 17} colab_type="code" executionInfo={"elapsed": 240, "status": "ok", "timestamp": 1524617921564, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="oZYSXoNNQJPV" outputId="4bef5e85-abd1-472d-dc02-8a0145696a28"
predictions.to_csv('sample_data/itunes-amazon/unlabeled_predictions.csv')

# + [markdown] colab_type="text" id="3Xo0ULsqQJPX"
# #### Getting predictions on labeled data
#
# You can also get predictions for labeled data such as validation data. To do so, you can simply call the `run_prediction` method passing the validation data as argument.

# + colab={"autoexec": {"startup": false, "wait_interval": 0}, "base_uri": "https://localhost:8080/", "height": 490} colab_type="code" executionInfo={"elapsed": 2330, "status": "ok", "timestamp": 1524617938346, "user": {"displayName": "", "photoUrl": "", "userId": ""}, "user_tz": 300} id="iXvu3exAQJPY" outputId="0491b0fc-0fcd-416e-a3b4-9360fe052c7f"
valid_predictions = model.run_prediction(validation, output_attributes=True)
valid_predictions.head()

StarcoderdataJupyterScriptsDedupFiltered