# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# + [markdown] jupyter={"outputs_hidden": true}
# # The Dataset and Metadata objects

# + [markdown] jupyter={"outputs_hidden": false} pycharm={"name": "#%%\n"}
# Both the **Dataset** and **Metadata** objects are the pilar to leverage YData's package features.
# - The **Dataset** object it's an abstraction of differente Python engines for handling data: 
#     - Dask: If you're looking for scalability and still prefer to keep it.
#     - Pandas: If you want to keep it as pythonic as possible.
#     - Numpy: If arrays are your thing.
#     
#     
# - The **Metadata** is the object that helps you extracting the main data from your dataset:
#     - The columns metadata: Both Variable and Data type (numerical, categorical, etc.)
#     - The data warnings: Checks for the presence of duplicates, variables with skewness, etc.
#     
# The **Metadata** object only works with a **Dataset** as an input. In this Notebook, we will be showing you features and capabilities of this objects and how to combine them with other pieces from YData's package offer.

# +
import pandas as pd

from ydata.dataset import Dataset
from ydata.metadata import Metadata

# +
data = pd.read_csv('diabetes.csv')

#Create the dataset object
dataset = Dataset(data)

#Getting some info from the Dataset
#Schema - Columns and variable types
print('\033[1m Dataset schema \033[0m')
print(dataset.schema)

#Nrows - Number of rows
print(dataset.nrows)

print("\n\033[1m Dataset shape - Number of training rows and columns for both training and holdout \033[0m")
print(dataset.shape(lazy_eval=False))
# -

# ### Extract the metadata from the Dataset

# +
#init the metadata object
metadata = Metadata()

#calculate the Metadata of a given Dataset
metadata(dataset)

#Getting the all metadata summary
print('\n\033[1mMetadata summary\033[0m')
print(metadata.summary)

for item, values in metadata.summary.items():
    print('\n\033[4m'+item+'\033[0m')
    print(values)
# -

## Setting the target variable
metadata.target='readmitted'

# ### Updating columns datatypes
# The automated inferences might not be totally correct in all the cases. For that reason, we always recommend to update the datatypes accordingly to the user understanding of the data. 
#
# The update can be done by column or for a group of columns.

print(metadata)

# In this particular example, the 'encounter_id' columns have been mistakenly identified as a numerical columns instead of an ID. The code snippet below shows how to change the datatypes:

# +
#Getting the all metadata summary
print('\n\033[1mChanging one column data type\033[0m')
metadata.columns = {'encounter_id': 'id'}

print(f"'encounter_id': {metadata.columns['encounter_id'].datatype.name}")

print('\n\033[1mChanging multiple columns data types\033[0m')
metadata.columns = {'patient_nbr': 'id',
                    'admission_type_id': 'id'}

print(f"'patient_nbr': {metadata.columns['patient_nbr'].datatype.name}")
print(f"'admission_type_id': {metadata.columns['admission_type_id'].datatype.name}")
# -

# ### Filtering metadata by columns
# For some activities the full metadata might not be needed and only a portion of the calls shall be considered for a certain activity (eg. data synthesis). The Metadata object allows the users to select only the needed columns, as per the example below.

# +
filtered_metadata = metadata['encounter_id', 'age', 'acarbose', 'readmitted']

print('\n\033[1mNew avaialble metadata\033[0m')
print(filtered_metadata)
# -

# ## YData connectors and the Metadata

import os
from pathlib import Path
from ydata.connectors import GCSConnector
from ydata.utils.formats import read_json

token = read_json('gcs_credentials.json')
conn = GCSConnector(project_id=token['project_id'], keyfile_dict=token)
data = conn.read_file('gs://ydata_testdata/tabular/diabetes/data.csv')

# The Ydata connectors already return an object of type Dataset.
diabetes_metadata = Metadata()
diabetes_metadata(data)

print(diabetes_metadata)

StarcoderdataJupyterScriptsDedupFiltered