<jupyter_start><jupyter_text>Data Retrieval for Machine Learning In this notebook, we retrieve MLB data from 1996-2017 that we will use to fit a machine learning model to. We decided that we wanted to see if we could find a model that would predict team wins even better than the Pythagorean Expectation does. After researching and examining baseball statistics, we decided to not only focus on team statistics as a whole, but pitcher statistics as well. The performance of the pitcher has a huge impact on the game results.<jupyter_code>import numpy as np
import pandas as pd
import requests
import plotly.offline as py
import matplotlib.pyplot as plt
from plotly.graph_objs import *
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import cross_val_score
from sklearn.cluster import KMeans
from bs4 import BeautifulSoup
%matplotlib inline
pd.set_option("max_r", 15)
py.init_notebook_mode(connected=True)<jupyter_output><empty_output><jupyter_text>For unknown reasons, the seventh and eighth tables that we need from the above URLs do not appear in the html returned from BeautifulSoup (it skips over them). Thus, instead of scraping from them like we did for the other datasets we need in the previous notebook, we are taking CSV's. It is also worth noting that this is the only website that has the advanced metrics we need for our calculations, which makes this situation even more of a shame.<jupyter_code># read in all the csvs
dfs = []
for i in range(1996, 2017):
    fname = "mlb%d.csv" % i
    df = pd.read_csv(fname)
    df["year"] = i
    dfs.append(df)
# concat all the team data into one
mlb = pd.concat(dfs).dropna()
mlb = mlb.sort_values(by=["Tm","year"])
mlb.head()
# read in pitching data
dfs = []
for i in range(1996, 2017):
    fname = "pitch%d.csv" % i
    df = pd.read_csv(fname)
    df["year"] = i
    dfs.append(df)
# combine pitching data 
pitch = pd.concat(dfs).dropna()
pitch = pitch.sort_values(by=["Tm","year"])
pitch.head()
mlb = mlb.reset_index().drop("index",axis=1)
pitch = pitch.reset_index().drop("index",axis=1)
# Ensure all overlapping columns have the same values.
overlapping_cols = ['Tm', 'W', 'L', 'G', 'year']
df = (mlb[overlapping_cols] == pitch[overlapping_cols])
df.all()
# Convert runs and runs allowed to yearly numbers from averages per game.
mlb['R'] = mlb['R'] * 162
mlb['RA'] = mlb['RA'] * 162
# Add all of the columns we need into one dataframe.
mlb.head()<jupyter_output><empty_output><jupyter_text>The variables that we felt would be most usuful for predicting wins with a machine learning model were:+ Runs: the total number of runs the team scored during the season+ Runs Allowed: the total number of runs that were scored against the team during the season+ Strength of Schedule: the difficulty or ease of a team's opponent as compared to other teams.+ ERA: the mean of the earned runs given up by a pitcher per nine innings pitched+ WHIP: walks hits per inning pitched+ FIP: field independent pitching+ SO: strike outs<jupyter_code># Create a dataframe for our training data.
machine_data = pd.DataFrame(mlb[['R',"RA","SOS",'W',"year"]])
machine_data['ERA'] = pitch['ERA']
machine_data['WHIP'] = pitch['WHIP']
machine_data['FIP'] = pitch['FIP']
machine_data['SO'] = pitch['SO']
machine_data.dtypes
machine_data.to_csv("training.csv", index=False)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup