<jupyter_start><jupyter_text>Random ForestRandom Forest is an ensemble tree-based predictive algorithm. In this case we will be using it for regression - we want to predict a continuous number, predicted points, for each player each game. It works by training many separate decision trees, each using a subset of the training data, and outputs the average prediction across all trees.Applying it to a time series problem, where metrics from recent time periods can be predicitve, requires us to add in window features (e.g. points scored last gameweek). These are created using the player_lag_features function from 00_fpl_features.<jupyter_code># add a bunch of player lag features
lag_train_df, team_lag_vars = team_lag_features(train_df, ['total_points'], ['all', 1, 2, 3, 4, 5, 10])
lag_train_df, player_lag_vars = player_lag_features(lag_train_df, ['total_points'], ['all', 1, 2, 3, 4, 5, 10])<jupyter_output><empty_output><jupyter_text>Similar to the simple model, we'll set the validation period to be gameweeks 20-25 of the 2019/20 season - the model will be trained on all data prior to that period. This time however, we'll be using some additional features: the season, gameweek, player position, home/away, and both teams, as well as all the lagging features we created above.<jupyter_code># set validaton point/length and categorical/continuous variables
valid_season = '2021'
valid_gw = 20
valid_len = 6
cat_vars = ['season', 'position', 'was_home', 'team', 'opponent_team']
cont_vars = ['gw']#, 'minutes']
dep_var = ['total_points']<jupyter_output><empty_output><jupyter_text>Some of the features have an order (2019/20 season is after 2019 season) whereas others do not (position). We can set this in the data where appropriate using an ordered category (e.g. 1617 < 1718 < 1819 < 1920 < 2021).<jupyter_code># we want to set gw and season as ordered categorical variables
# need lists with ordered categories
ordered_gws = list(range(1,39))
ordered_seasons = ['1617', '1718', '1819', '1920', '2021']

# set as categories with correct order 
lag_train_df['gw'] = lag_train_df['gw'].astype('category')
lag_train_df['season'] = lag_train_df['season'].astype('category')

lag_train_df['gw'].cat.set_categories(ordered_gws, ordered=True, inplace=True)
lag_train_df['season'].cat.set_categories(ordered_seasons, ordered=True, inplace=True)
lag_train_df['season']<jupyter_output><empty_output><jupyter_text>And now we can go ahead and create our training and validation sets using the function we defined in the last notebook.<jupyter_code># create dataset with adjusted post-validation lag numbers
train_valid_df, train_idx, valid_idx = create_lag_train(lag_train_df, 
                                                      cat_vars, cont_vars, 
                                                      player_lag_vars, team_lag_vars, dep_var,
                                                      valid_season, valid_gw, valid_len)<jupyter_output><empty_output><jupyter_text>The way we calculate our lag features means that there will be null values in our dataset. This will cause an error when using random forest in scikit learn, so we will set them all to zero for now (although note that this may not be the best fill strategy).<jupyter_code>lag_train_df[~np.isfinite(lag_train_df['total_points_pg_last_1'])]
# imp = SimpleImputer(missing_values=np.nan, strategy='mean')
# need to think about imputing NaN instead of setting to zero
# imp.fit(X_train[team_lag_vars + player_lag_vars])
train_valid_df[team_lag_vars + player_lag_vars] = train_valid_df[team_lag_vars + player_lag_vars].fillna(0)<jupyter_output><empty_output><jupyter_text>The random forest regressor will only take numbers as inputs, so we need to transform our caterogical features into a format that the random forest regressor object will be able to use, numbers instead of strings in one or more columns.<jupyter_code># split out dependent variable
X, y = train_valid_df[cat_vars + cont_vars + team_lag_vars + player_lag_vars].copy(), train_valid_df[dep_var].copy()
# since position is categorical, it should be a string
X['position'] = X['position'].apply(str)

# need to transform season
enc = LabelEncoder()
X['season'] = enc.fit_transform(X['season'])
X_dict = X.to_dict("records")

# Create the DictVectorizer object: dv
dv = DictVectorizer(sparse=False, separator='_')

# Apply dv on df: df_encoded
X_encoded = dv.fit_transform(X_dict)

X_df = pd.DataFrame(X_encoded, columns=dv.feature_names_)<jupyter_output><empty_output><jupyter_text>For example, season is now represented by a number (0 -> 2016/17, 1 -> 2017/18, etc.) in a single column, and position is represented by a 1 or 0 in multiple columns.<jupyter_code>X_df[['season', 'position_1', 'position_2', 'position_3', 'position_4']]
X_df.columns<jupyter_output><empty_output><jupyter_text>Let's now split out our training (everything prior to the validation gameweek) and validation (6 gameweeks from the validation gameweek, only rows with >0 minutes)<jupyter_code># split out training and validation sets
X_train = X_df.loc[train_idx]
y_train = y.loc[train_idx]
X_test = X_df.loc[valid_idx]
# we only want look at rows with >0 minutes (i.e. the player played)
# test_mask = (X_test['minutes'] > 0)
# X_test = X_test[test_mask]
# y_test = y.loc[valid_idx][test_mask]
y_test = y.loc[valid_idx]
# X_train = X_train.drop('minutes', axis=1)
# X_test = X_test.drop('minutes', axis=1)<jupyter_output><empty_output><jupyter_text>We can now create the RandomForestRegessor with set parameters, train using the training data, and look at the error on the validation set.<jupyter_code># def rf(xs, y, n_estimators=40, max_samples=50_000,
#        max_features=0.5, min_samples_leaf=5, **kwargs):
#     return RandomForestRegressor(n_jobs=-1, n_estimators=n_estimators,
#         max_samples=max_samples, max_features=max_features,
#         min_samples_leaf=min_samples_leaf, oob_score=True).fit(xs, y)
def rf(xs, y, max_depth=7, **kwargs):
    return RandomForestRegressor(n_jobs=-1, max_depth=max_depth, oob_score=True).fit(xs, y)
# fit training data
m = rf(X_train, y_train.values.ravel())
# predict validation set and output metrics
preds = m.predict(X_test)
print("RMSE: %f" % (r_mse(preds, y_test.values.ravel())))
print("MAE: %f" % mae(preds, y_test.values.ravel()))<jupyter_output>RMSE: 2.147401
MAE: 1.217610<jupyter_text>Right away this looks like it's a significant improvement on the simple model, good to see. Let's go ahead and use the same approach with validation across the whole of the 2019/20 season.<jupyter_code>def rf_season(df, valid_season='2021'):
    # empty list for scores
    scores = []
    valid_len = 6
    
    for valid_gw in range(1,40-valid_len):
        # create dataset with adjusted post-validation lag numbers
        train_valid_df, train_idx, valid_idx = create_lag_train(df, cat_vars, cont_vars, 
                                                                player_lag_vars, team_lag_vars, dep_var,
                                                                valid_season, valid_gw, valid_len)
        
        train_valid_df[team_lag_vars + player_lag_vars] = train_valid_df[team_lag_vars + player_lag_vars].fillna(0)
        
        # split out dependent variable
        X, y = train_valid_df[cat_vars + cont_vars + team_lag_vars + player_lag_vars].copy(), train_valid_df[dep_var].copy()
        
        # since position is categorical, it should be a string
        X['position'] = X['position'].apply(str)

        # need to transform season
        enc = LabelEncoder()
        X['season'] = enc.fit_transform(X['season'])
        X_dict = X.to_dict("records")

        # Create the DictVectorizer object: dv
        dv = DictVectorizer(sparse=False, separator='_')

        # Apply dv on df: df_encoded
        X_encoded = dv.fit_transform(X_dict)
        X_df = pd.DataFrame(X_encoded, columns=dv.feature_names_)
        
        # split out training and validation sets
        X_train = X_df.loc[train_idx]
        y_train = y.loc[train_idx]
        X_test = X_df.loc[valid_idx]
        # we only want look at rows with >0 minutes (i.e. the player played)
#         test_mask = (X_test['minutes'] > 0)
#         X_test = X_test[test_mask]
#         y_test = y.loc[valid_idx][test_mask]
        y_test = y.loc[valid_idx]
        
        m = rf(X_train, y_train.values.ravel()) 
        preds, targs = m.predict(X_test), y_test.values.ravel()
        gw_mae = mae(preds, targs)
        print("GW%d MAE: %f" % (valid_gw, gw_mae))
        
        scores.append(gw_mae)
        
    return scores
scores = rf_season(lag_train_df)
plt.plot(scores)
plt.ylabel('GW MAE')
plt.xlabel('GW')
plt.text(15, 1.55, 'Season Avg MAE: %.2f' % np.mean(scores), bbox={'facecolor':'white', 'alpha':1, 'pad':5})
plt.show()<jupyter_output><empty_output><jupyter_text>Looking across the whole season we see about a 10% improvement versus the simple model. Also interesting is that the performance again improves as the season progresses - this makes sense, more data about each of teams and players (particularly new ones) means improved ability to predict the next 6 gameweeks.Let's add these validation scores to our comparison dataset.<jupyter_code>model_validation_scores = pd.read_csv(path/'charts/model_validation_scores.csv', index_col=0)
model_validation_scores['random_forest'] = scores
model_validation_scores.to_csv(path/'charts/model_validation_scores.csv')<jupyter_output><empty_output><jupyter_text>A feature of the random forest algorithm is that we can see how often features are being used in trees. This will give us an indication of how important each feature is i.e. is it predictive of todal points scored. Simple models are usually better, so this also gives us a way of seeing if there are any features that are not particularly useful, and can therefore be removed.<jupyter_code>def rf_feat_importance(m, df):
    return pd.DataFrame({'cols':df.columns, 'imp':m.feature_importances_}
                       ).sort_values('imp', ascending=False)
fi = rf_feat_importance(m, X_train)
fi[:32]
def plot_fi(fi):
    return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False).invert_yaxis()

plot_fi(fi[:30]);<jupyter_output><empty_output><jupyter_text>At the moment this algorithm is given minutes played in the gameweek so it's unsurprising that this is by far the most important feature - the more minutes a player plays, the more opportunity to score points. But strictly speaking we don't actually have this information prior to a gameweek (in practice it is estimated using previous minutes and injury status), so we can ignore it for now.Below that the top features are:1. minutes_last_1 - number of minutes in the last fixture for the player2. minutes_last_2 - number of minutes in the last two fixtures for the player3. total_points_pg_last_all - the player's average points per game in all of history (since start of 2016/17 season)4. total_points_team_pg_last_all_opponent - the opposition's average points per game in all of history5. minutes_last_3 - number of minutes in the last three fixtures for the player6. total_points_team_pg_last_all - the player's team's average points per game in all of history7. total_points_pg_last_10 - the player's average points per game in the last 10 fixtures8. total_points_pg_last_1 - the player's average points per game in the last fixtureThis is interesting. It seems to be saying that the amount of minutes a player has played recently and their underlying ability to score points in all of history, along with their team's and opponent team's points scoring in all of history, is most important.Recent performance (i.e. 'form') is also important, but to a lesser extent.It also shows that the lag features are far more useful than the categorical features such as team, opponent and position. Again not too surprising since information on these categories are already captured in the lag features.Let's test this... we can remove anything with a feature importance of less than 0.005 and see how the model performs on the original 2019/20 week 20 validation point (going from 94 features to just 32).<jupyter_code>to_keep = fi[fi.imp>0.005].cols
len(to_keep)
len(X_train.columns)
X_train_imp = X_train[to_keep]
X_test_imp = X_test[to_keep]
m = rf(X_train_imp, y_train.values.ravel())
mae(m.predict(X_test_imp), y_test.values.ravel())
# mae(m.predict(X_train_imp), y_train.values.ravel())<jupyter_output><empty_output><jupyter_text>Very similar albeit slightly higher error (less than 1% worse performance) than previously, and still a long way ahead of the simple model.Continuing our thinking about improving/simplifying the model features, we can also look to see if there are any similar features - quite often we will find that some features are so similar that some of them may be redundant.The following function determines the similarity between columns in a dataset and visualises it using a dendrogram.<jupyter_code>def cluster_columns(df, figsize=(10,6), font_size=12):
    corr = np.round(sp.stats.spearmanr(df).correlation, 4)
    corr_condensed = hc.distance.squareform(1-corr)
    z = hc.linkage(corr_condensed, method='average')
    fig = plt.figure(figsize=figsize)
    hc.dendrogram(z, labels=df.columns, orientation='left', leaf_font_size=font_size)
    plt.show()
cluster_columns(X_train_imp)<jupyter_output><empty_output><jupyter_text>We can see that our lagging features are somewhat similar - absolutely expected since, for example, minutes_last_5 is equal to minutes_last_4 + minutes 5 games ago. They are still different enough to be of value separately, but it does make me wonder whether separating out each historic game in some way (up to a point) would be valuable. A final useful tool we can use is partial dependency plots. These try to look at the impact of single features on the dependent variable (points scored).<jupyter_code>fig,ax = plt.subplots(figsize=(12, 3))
plot_partial_dependence(m, X_test_imp, ['total_points_pg_last_all', 
                                    'total_points_team_pg_last_all_opponent',
                                    'total_points_pg_last_1'],
                        grid_resolution=20, ax=ax);<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup