<jupyter_start><jupyter_text>Predicting a customer's next purchase using automated feature engineering    **As customers use your product, they leave behind a trail of behaviors that indicate how they will act in the future. Through automated feature engineering we can identify the predictive patterns in granular customer behavioral data that can be used to improve the customer's experience and generate additional revenue for your business.**In this tutorial, we show how [Featuretools](www.featuretools.com) can be used to perform feature engineering on a multi-table dataset of 3 million online grocery orders provided by Instacart to train an accurate machine learning model to predict what product a customer buys next.*Note: If you are running this notebook yourself, refer to the [read me on Github](https://github.com/featuretools/predict_next_purchaserunning-the-tutorial) for instructions to download the Instacart dataset* Highlights* We automatically generate 150+ features using Deep Feature Synthesis and select the 20 most important features for predictive modeling* We build a pipeline that it can be reused for numerous prediction problems (you can try this yourself!)* We quickly develop a model on a subset of the data and validate on the entire dataset in a scalable manner using [Dask](http://dask.pydata.org/en/latest/).<jupyter_code>import featuretools as ft
from dask import bag
from dask.diagnostics import ProgressBar
import pandas as pd
import utils
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import os
ft.__version__<jupyter_output><empty_output><jupyter_text>Step 1. Load dataWe start by loading in just one partion of our datast. In this case, a partition of our dataset contains the complete purchase history for each user within it. At the end of the tutorial, we will run the pipeline on every partiton to generate a final model. To learn more about loading data into Featuretools, read the guide [here](https://docs.featuretools.com/loading_data/using_entitysets.html).<jupyter_code>es = utils.load_entityset("partitioned_data/part_1/")
es<jupyter_output><empty_output><jupyter_text>Step 2. Make LabelsFor supervised machine learning, we need labels. These labels define what our predictive model will be used for. In this tutorial, we will predict if a customer will buy Bananas in the next 4 weeks.We generate training examples by selecting a `cutoff_time` in the past to make our labels. Using users who had acivity during `training_window` days before the `cutoff_time`, we look to see if they purchase the product in the `prediction_window`. If you are running this code yourself, feel free to experiment with any of these parameters! For example, try to predict if a customer will buy "Limes" instead of "Bananas" or increase the size of your `prediction_window`.<jupyter_code>label_times = utils.make_labels(es=es,
                                product_name = "Banana",
                                cutoff_time = pd.Timestamp('March 15, 2015'),
                                prediction_window = ft.Timedelta("4 weeks"),
                                training_window = ft.Timedelta("60 days"))
label_times.head(5)<jupyter_output><empty_output><jupyter_text>We can see above the our training examples contain three pieces of information: a user id, the last time we can use data before feature engineering (called the "cutoff time"), and the label to predict. These are called our "label times". The distribution of the labels<jupyter_code>label_times["label"].value_counts()<jupyter_output><empty_output><jupyter_text>3. Automated Feature EngineeringWith our label times in hand, we can use Deep Feature Synthesis to automatically generate features.When we use DFS, we specify* `target_entity` - the table to build feature for* `cutoff_time` the point in time to calculate the features* `training_window` - the amount of historical data we want to use when calculating featuresA good way to think of the `cutoff_time` is that it let's us "pretend" we are at an earlier point in time when generating our features so we can simulate making predictions. We get this time for each customer from the label times we generated above.<jupyter_code>feature_matrix, features = ft.dfs(target_entity="users", 
                                  cutoff_time=label_times,
                                  training_window=ft.Timedelta("60 days"), # same as above
                                  entityset=es,
                                  verbose=True)
# encode categorical values
fm_encoded, features_encoded = ft.encode_features(feature_matrix,
                                                  features)

print "Number of features %s" % len(features_encoded)
fm_encoded.head(10)<jupyter_output>Building features: 120it [00:00, 6144.38it/s]
Progress: 100%|██████████| 1/1 [00:09<00:00,  9.61s/cutoff time]
Number of features 161<jupyter_text>Step 4. Machine LearningUsing the default parameters, we generated 160 potential features for our prediction problem. With a few simple commands, this feature matrix can be used for machine learning<jupyter_code>X = utils.merge_features_labels(fm_encoded, label_times)
X.drop(["user_id", "time"], axis=1, inplace=True)
X = X.fillna(0)
y = X.pop("label")<jupyter_output><empty_output><jupyter_text>Let's train a Random Forest and validate using 3-fold cross validation<jupyter_code>clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)
scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,
                         scoring="roc_auc", verbose=True)

"AUC %.2f +/- %.2f" % (scores.mean(), scores.std())<jupyter_output>[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    4.6s finished<jupyter_text>We can see we perform noticably better than guessing! However, we have a pretty high difference in performance accross folds.To improve this, let's identify the top 20 features using a Random Forest and then perform machine learning on the whole dataset (all of the partions).<jupyter_code>clf.fit(X, y)
top_features = utils.feature_importances(clf, features_encoded, n=20)<jupyter_output>1: Feature: COUNT(order_products WHERE product_name = Banana), 0.129
2: Feature: MODE(order_products.product_name) = Banana, 0.046
3: Feature: MODE(orders.MODE(order_products.product_name)) = Banana, 0.025
4: Feature: MEAN(orders.COUNT(order_products)), 0.020
5: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.019
6: Feature: COUNT(order_products WHERE department = produce), 0.019
7: Feature: COUNT(order_products WHERE department = dairy eggs), 0.019
8: Feature: STD(orders.NUM_UNIQUE(order_products.department)), 0.018
9: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.017
10: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.017
11: Feature: MEAN(orders.PERCENT_TRUE(order_products.reordered)), 0.017
12: Feature: PERCENT_TRUE(order_products.reordered), 0.016
13: Feature: STD(orders.COUNT(order_products)), 0.016
14: Feature: COUNT(order_products), 0.016
15: Feature: NUM_UNIQUE(order_products.product_name), 0.015
16: Feature: SKEW(orders.NUM_UNIQU[...]<jupyter_text>To persist this features, we can save them to disk.<jupyter_code>ft.save_features(top_features, "top_features")<jupyter_output><empty_output><jupyter_text>Understanding feature engineering in Featuretools Before moving forward, take a look at the feature we created. You will see that they are more than just simple transformations of columns in our raw data. Instead, they aggregations (and sometimes stacking of aggregations) across the relationships in our dataset. If you're curious how this works, learn about the Deep Feature Synthesis algorithm in our documentation [here](https://docs.featuretools.com/automated_feature_engineering/afe.html).DFS is so powerful because with no manual work, the library figured out that historical purchases of bananas are important for predicting future purchases. Additionally, it surfaces that purchasing dairy or eggs and reordering behavior are important features. Even though these features are intuitive, Deep Feature Synthesis will automatically adapt as we change the prediction problem, saving us the time of manually brainstorming and implementing these data transformation.   Scaling to full datasetOnce we have written the pipeline for one partition, we can easily scale it out to the full dataset using [Dask](dask.pydata.org). A similar pipeline could also be built using [Spark](http://spark.apache.org/docs/2.2.0/api/python/).<jupyter_code>pbar = ProgressBar()
pbar.register()<jupyter_output><empty_output><jupyter_text>First, we assemble our partitions and map them to entity sets using the function from before. A single partition contains all the data for each user within it, so this computation is easily parallelized.<jupyter_code>path = "partitioned_data/"
_, dirnames, _ = os.walk(path).next()
dirnames = [path+d for d in dirnames]
b = bag.from_sequence(dirnames)
entity_sets = b.map(utils.load_entityset)<jupyter_output><empty_output><jupyter_text>Next, we create label times for each entity set<jupyter_code>label_times = entity_sets.map(utils.dask_make_labels,
                              product_name = "Banana",
                              cutoff_time = pd.Timestamp('March 1, 2015'),
                              prediction_window = ft.Timedelta("4 weeks"),
                              training_window = ft.Timedelta("60 days"))
label_times
# load in the features from before
top_features = ft.load_features("top_features", es)
feature_matrices = label_times.map(utils.calculate_feature_matrix, features=top_features)<jupyter_output><empty_output><jupyter_text>Now, we compute with Dask. Running on a Macbook Pro with a 2.2 GHz Intel Core i7 and 16gb of ram, this takes about 20 minutes to run.<jupyter_code>fms_out = feature_matrices.compute()
X = pd.concat(fms_out)<jupyter_output>[########################################] | 100% Completed | 20min 31.3s<jupyter_text>Now, we repeat the same machine learning steps from the sample dataset<jupyter_code>X.drop(["user_id", "time"], axis=1, inplace=True)
y = X.pop("label")
clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)
scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,
                         scoring="roc_auc", verbose=True)

"AUC %.2f +/- %.2f" % (scores.mean(), scores.std())<jupyter_output>[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:  1.5min finished<jupyter_text>We can now we that our accuracy has stabalized across folds, giving us much more confidence in our model.Now, let's look at the top features<jupyter_code>clf.fit(X, y)
top_features = utils.feature_importances(clf, top_features, n=20)<jupyter_output>1: Feature: COUNT(order_products WHERE product_name = Banana), 0.272
2: Feature: MODE(order_products.product_name) = Banana, 0.065
3: Feature: MODE(orders.MODE(order_products.product_name)) = Banana, 0.045
4: Feature: STD(orders.COUNT(order_products)), 0.043
5: Feature: COUNT(order_products WHERE department = produce), 0.042
6: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.040
7: Feature: MEAN(orders.PERCENT_TRUE(order_products.reordered)), 0.040
8: Feature: PERCENT_TRUE(order_products.reordered), 0.040
9: Feature: SUM(orders.PERCENT_TRUE(order_products.reordered)), 0.039
10: Feature: STD(orders.NUM_UNIQUE(order_products.department)), 0.037
11: Feature: SKEW(orders.PERCENT_TRUE(order_products.reordered)), 0.036
12: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.036
13: Feature: MEAN(orders.NUM_UNIQUE(order_products.department)), 0.035
14: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.034
15: Feature: MEAN(orders.COUNT(order_products)), [...]