<jupyter_start><jupyter_text>Scikit-Learn-style APIThis example demontrates compatability with scikit-learn's basic `fit` API.For demonstration, we'll use the perennial NYC taxi cab dataset.<jupyter_code>import os
import s3fs
import pandas as pd
import dask.array as da
import dask.dataframe as dd
from distributed import Client

from dask import persist
from dask_glm.estimators import LogisticRegression
if not os.path.exists('trip.csv'):
    s3 = s3fs.S3FileSystem(anon=True)
    s3.get("dask-data/nyc-taxi/2015/yellow_tripdata_2015-01.csv", "trip.csv")
client = Client()
ddf = dd.read_csv("trip.csv")<jupyter_output><empty_output><jupyter_text>We can use the `dask.dataframe` API to explore the dataset, and notice that some of the values look suspicious:<jupyter_code>ddf[['trip_distance', 'fare_amount']].describe().compute()<jupyter_output><empty_output><jupyter_text>Scikit-learn doesn't currently support filtering observations inside a pipeline ([yet](https://github.com/scikit-learn/scikit-learn/issues/3855)), so we'll do this before anything else.<jupyter_code># these filter out less than 1% of the observations
ddf = ddf[(ddf.trip_distance < 20) &
          (ddf.fare_amount < 150)]<jupyter_output><empty_output><jupyter_text>Now, we'll split our DataFrame into a train and test set, and select our feature matrix and target column (whether the passenger tipped). To ensure this example runs quickly for the documentation, we'll make the training smaller than usual.<jupyter_code>df_train, df_test = ddf.random_split([0.05, 0.95], random_state=2)

columns = ['VendorID', 'passenger_count', 'trip_distance', 'payment_type', 'fare_amount']

X_train, y_train = df_train[columns], df_train['tip_amount'] > 0
X_test, y_test = df_test[columns], df_test['tip_amount'] > 0

X_train = X_train.repartition(npartitions=2)
y_train = y_train.repartition(npartitions=2)

X_train, y_train, X_test, y_test = persist(
    X_train, y_train, X_test, y_test
)<jupyter_output><empty_output><jupyter_text>With our training data in hand, we fit our logistic regression.Nothing here should be surprising to those familiar with `scikit-learn`.<jupyter_code>%%time
# this is a *dask-glm* LogisticRegresion, not scikit-learn
lm = LogisticRegression(fit_intercept=False)
lm.fit(X_train.values, y_train.values)<jupyter_output>CPU times: user 4.99 s, sys: 1.48 s, total: 6.47 s
Wall time: 57.7 s<jupyter_text>Again, following the lead of scikit-learn we can measure the performance of the estimator on the training dataset:<jupyter_code>lm.score(X_train.values, y_train.values).compute()<jupyter_output><empty_output><jupyter_text>and on the test dataset:<jupyter_code>lm.score(X_test.values, y_test.values).compute()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup