# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: 'Python 3.7.9 64-bit (''teste'': conda)'
#     language: python
#     name: python37964bittesteconda1543f1c219704c4ea077a4e6376e3e28
# ---

# <img src="https://jaipresentation.blob.core.windows.net/comm/jai_avatar.png" width="100" align="right"/>
#
# # JAI - Trust your data
#
# ## Resolution: finding duplicates in your data
# This is an example of how to use the resolution capabilities of JAI.
#
# In this notebook we will use a subset of the [PC Games 2020](https://www.kaggle.com/jesneuman/pc-games) dataset to search for duplicate values in the 'Genres' column.
#
# You can install JAI in your environment using `pip install jai-sdk`.
#
# And you can read the docs [here](https://jai-sdk.readthedocs.io/en/stable/)!
#
# If you have any comments or suggestions, feel free to contact us: <EMAIL>
#
# *In no direction that we turn do we find ease or comfort. If we are honest and if we have the will to win we find only danger, hard work and iron resolution.* - <NAME>

# +
# JAI imports
from jai import Jai
from jai.processing import process_resolution

# I/O import
import pandas as pd
# -

# ## Reading data

# it might take a few seconds to download this dataset (10MB) to your computer
DATASET_URL = "https://jaipresentation.blob.core.windows.net/data/games_jai.parquet"
df_games = pd.read_parquet(DATASET_URL)

# checking values in the Platform column
df_games["Platform"].value_counts()

# We can see column `Platform` has some values that actually refer to the same thing (i.e., "Linux, macOS, PC" and "Linux, PC, macOS"). In other words, these are duplicate values.

# ## We can use JAI to find duplicates and standardize their values!

j = Jai("YOUR_AUTH_KEY")

# ### We call `resolution` passing a given `name` for the database and the `data` itself (i.e., column "Platform")

db_name = "games_resolution"
col = "Platform"
results = j.resolution(name=db_name, data=df_games[col], db_type="FastText")

# ## OK, how do I interpret these results?
# Each index in the `results` dataframe is related to an integer in the `resolution_id` column. This integer, in turn, is also an index! And it indicates which other sample is related to that index. So index 0 points to `resolution_id` 0, stating that sample number 0 is related to itself. No surprises there.

results.head(10)

# But what about indexes that are related to samples other than themselves? This will clear things up. Let's look at samples where the index DOES NOT match the `resolution_id` column:

res2 = results.copy()
res2["map_id"] = res2.index
res3 = res2.loc[res2["resolution_id"] != res2["map_id"]]
res3

# **Now we can see that samples 5 and 15 are actually referring to the same thing! Let's see if it checks out:**

print(f"Item 5: {df_games[col].iloc[5]}\nItem 15: {df_games[col].iloc[15]}")

# It does check out! These samples are clearly a permutation of one another. 

# ## We can create groups of samples that refer to the same thing
# This makes it easier for us to check if the output of `fill` is actually making any sense

# get groups
groups = dict()
for i in range(res3.shape[0]):
    fixed = res3["resolution_id"].iloc[i]
    moving = res3["map_id"].iloc[i]

    fixed_name = df_games[col].iloc[fixed]
    moving_name = df_games[col].iloc[moving]

    if fixed_name not in groups:
        groups[fixed_name] = {moving_name}
    else:
        groups[fixed_name].add(moving_name)

# get all platforms that correspond to 'Linux, PC, macOS, PlayStation 4' 
groups["Linux, PC, macOS, PlayStation 4"]

# The cell above shows all values that belong to the same group (`"Linux, PC, macOS, PlayStation 4"`). They are all permutations of one another and are indeed duplicates!
#
# Finally, we can standardize column `Platform` mapping permutations to a single, consistent value using the `results` dataframe.

results.head(10)

new_platform = [df_games[col].iloc[results["resolution_id"].iloc[item]] for item in range(results.shape[0])]

new_platform_col = "Platform resolved"
df_games_resolved = df_games.copy()
df_games_resolved[new_platform_col] = new_platform

print(f"[df_games]\nItem 5: {df_games[col].iloc[5]}\nItem 15: {df_games[col].iloc[15]}")
print(f"\n[df_games_resolved]\nItem 5: {df_games_resolved[new_platform_col].iloc[5]}\nItem 15: {df_games_resolved[new_platform_col].iloc[15]}")

StarcoderdataJupyterScriptsDedupFiltered