<jupyter_start><jupyter_text>CPE 695 Final Project Team 1 Data Descriptionhttps://archive.ics.uci.edu/ml/datasets/Audit+Data<jupyter_code># Data Filepath
data_filepath = 'audit_data'
# Packages
import pandas as pd
import random
import warnings
from IPython.display import Image, HTML, display
import pydotplus
import seaborn as sns
import matplotlib.pyplot as plt

# Sklearn
import sklearn
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix
from sklearn.ensemble import RandomForestClassifier

import helper

warnings.filterwarnings("ignore")
plt.style.use('fivethirtyeight')

# Load Data
df_audit = pd.read_csv('/'.join([data_filepath, 'audit_risk.csv']))
df_trial = pd.read_csv('/'.join([data_filepath, 'trial.csv']))<jupyter_output><empty_output><jupyter_text>Summarize Data  Audit Data (Training Data)<jupyter_code># Summarize 
print(f' Aduit shape: {df_audit.shape}')
display(df_audit.head())
display(df_audit.tail())
display(df_audit.info())
df_audit.describe()
df_audit.nunique()<jupyter_output><empty_output><jupyter_text>Trial Data (Test Data)<jupyter_code># Summarize 
print(f'Trial shape: {df_audit.shape}')
display(df_trial.head())
display(df_trial.tail())
display(df_trial.info())
df_trial.describe()
df_trial.nunique()<jupyter_output><empty_output><jupyter_text>Select Data: TrialThere are two datasets included in the file audit and trial. The two datasets contain the same data, but the audit data is far more expansive. The audit data contains raw data (Para A, Para B, Money Value, etc.) and formulas (Score A, Score B,  Score MV, etc.).  Comptroller and Auditor General of India (CAG) derived the formulas. The trial data has fewer formulaic calculations. The purpose of this project is to create a predictive model based on the raw data. The project will use the trial data and remove the calculated scores.  The model will overfit the data if the formulas are included.  Clean Data<jupyter_code># Variables
string_cols = ['LOCATION_ID']
dep_col = 'Risk'
dep_col_idx = -1
drop_cols = ['Score']
cols_keep = ['Sector_score', 'LOCATION_ID', 'PARA_A', 'PARA_B', 'numbers', 
             'Money_Value', 'District_Loss', 'PROB', 'History', 'Prob', 'Risk']

cols_keep = ['Sector_score', 'LOCATION_ID', 'PARA_A', 'PARA_B', 
             'Money_Value', 'District_Loss', 'History', 'Risk']
df_audit.columns
df = df_audit.copy()
df = df.loc[:, cols_keep]
df = helper.fill_nas_df(df)

# No Location
df_no_location = df.copy().drop(columns= string_cols)

# Location Dummy Only
df_location_dummy = pd.get_dummies(df.loc[:, string_cols])

# Location Dummy
df_location_included = helper.dummy_string_var(df, string_cols)<jupyter_output><empty_output><jupyter_text>Ignore Locations and drop columns<jupyter_code>df = df_no_location.copy()
#df = df.drop(columns = drop_cols)

df.head()
# Split
X = df.drop(columns = [dep_col])
y = df.loc[:, dep_col]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup