<jupyter_start><jupyter_text>Clean column names (delete spaces, special characters, ...)<jupyter_code>params = preprocessing.params
dfs = preprocessing._transform_column_names(INPUT_FILE_PATH, OUTPUT_FILE_PATH)
dfs['ingredients'].head()<jupyter_output><empty_output><jupyter_text>Generate a dataframe with statistics of ingredients<jupyter_code>orig_ing_df = dfs['ingredients']
desc_ingred_df = preprocessing._get_describe_ingredients(orig_ing_df)
print(desc_ingred_df.shape)
desc_ingred_df.head(5)<jupyter_output>(213, 3)<jupyter_text>Merge original ingredients dataset with statistics (min, max)<jupyter_code>ing_df = orig_ing_df.merge(desc_ingred_df, how='left', left_on='Ingredients', right_on='Ingredients') 
print(ing_df.shape)
ing_df.head(5)<jupyter_output>(5671, 9)<jupyter_text>Create sparse matrix with ingredients<jupyter_code>key_columns = ['Ciqual_AGB', 'Nom_Francais', 'Groupe_aliment', 'Sous-groupe_aliment', 'LCI_Name' , 'Ingredients']
pivot_column = 'Ingredients'
value_columns = 'Score_unique_EF_'
drop_cols = ['Nom_Francais', 'Groupe_aliment', 'Sous-groupe_aliment', 'LCI_Name']
sparse_ing_df = preprocessing._create_sparse_ingredients(ing_df,key_columns, pivot_column, value_columns)
sparse_ing_df.head(3)<jupyter_output><empty_output><jupyter_text>Create sparse matrix where min_EF, max_EF of each ingredient are in columns<jupyter_code>pivot_index = ['Ciqual_AGB', 'Nom_Francais', 'Groupe_aliment', 'Sous-groupe_aliment', 'LCI_Name']
pivot_columns = ['Ingredients']
pivot_values = ['min_EF', 'max_EF']
drop_cols = ['Nom_Francais', 'Groupe_aliment', 'Sous-groupe_aliment', 'LCI_Name']
pivot_ing_minmax = preprocessing._create_features_min_max_ing(ing_df, pivot_index, pivot_columns, pivot_values)
pivot_ing_minmax.head()<jupyter_output><empty_output><jupyter_text>Merge sparse ingredients data with pivoted min max values<jupyter_code>drop_cols = ['Nom_Francais', 'Groupe_aliment', 'Sous-groupe_aliment', 'LCI_Name']
new_ing_df = sparse_ing_df.merge(pivot_ing_minmax, how = 'inner', left_on='Ciqual_AGB', right_on="Ciqual_AGB")
dfs['ingredients'] = new_ing_df
new_ing_df.head(5)
synthese_key = params.synthese.index_key.to_list()#get_param('synthese', 'index_key')

ingredients_key = params.ingredients.index_key.to_list() #get_param('ingredients', 'index_key')
etapes_key = params.etapes.index_key.to_list()#get_param('etapes', 'index_key')

new_ingred_df = preprocessing.merge_dataset(dfs['synthese'], dfs['ingredients'], synthese_key,ingredients_key)
new_etape_df = preprocessing.merge_dataset(dfs['synthese'], dfs['etapes'], synthese_key, etapes_key)
new_ingred_df.head(5)<jupyter_output><empty_output><jupyter_text>Clean nan values and save the cleaned files in OUPUT_FILE_PATH<jupyter_code>new_ingred_df.fillna(0, inplace=True)
new_etape_df.fillna(0, inplace=True)

OUTPUT_FILE_PATH = './../data/processed'
new_ingred_df.to_csv(os.path.join(OUTPUT_FILE_PATH, params.ingredients.file_name), index=False)
new_etape_df.to_csv(os.path.join(OUTPUT_FILE_PATH, params.etapes.file_name), index=False)
new_ingred_df<jupyter_output><empty_output>