<jupyter_start><jupyter_text>Introduction to cuDF You will begin your accelerated data science training with an introduction to [cuDF](https://github.com/rapidsai/cudf), the RAPIDS API that enables you to create and manipulate GPU-accelerated dataframes. cuDF implements a very similar interface to Pandas so that Python data scientists can use it with very little ramp up. Throughout this notebook we will provide Pandas counterparts to the cuDF operations you perform to build your intuition about how much faster cuDF can be, even for seemingly simple operations.  Objectives By the time you complete this notebook you will be able to:- Read and write data to and from disk with cuDF- Perform basic data exploration and cleaning operations with cuDF  Imports Here we import cuDF and CuPy for GPU-accelerated dataframes and math operations, plus the CPU libraries Pandas and NumPy on which they are based and which we will use for performance comparisons:<jupyter_code>import cudf
import cupy as cp

import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Reading and Writing Data Using [cuDF](https://github.com/rapidsai/cudf), the RAPIDS API providing a GPU-accelerated dataframe, we can read data from [a variety of formats](https://rapidsai.github.io/projects/cudf/en/0.10.0/api.htmlmodule-cudf.io.csv), including csv, json, parquet, feather, orc, and Pandas dataframes, among others.For the first part of this workshop, we will be reading almost 60 million records (corresponding to the entire population of England and Wales) which were synthesized from official UK census data. Here we read this data from a local csv file directly into GPU memory:<jupyter_code>%time gdf = cudf.read_csv('../data/data_pop.csv')
gdf.shape
gdf.drop(gdf.columns[0], axis=1, inplace=True)
gdf.dtypes<jupyter_output><empty_output><jupyter_text>Here for comparison we read the same data into a Pandas dataframe:<jupyter_code>%time df = pd.read_csv('../data/data_pop.csv')
df.drop(df.columns[0], axis=1, inplace=True)
gdf.shape == df.shape<jupyter_output><empty_output><jupyter_text>Because of the sophisticated GPU memory management behind the scenes in cuDF, the first data load into a fresh RAPIDS memory environment is sometimes substantially slower than subsequent loads. The RAPIDS Memory Manager is preparing additional memory to accommodate the array of data science operations that you may be interested in using on the data, rather than allocating and deallocating the memory repeatedly throughout your workflow.We will be using `gdf` regularly in this workshop to represent a GPU dataframe, as well as `df` for a CPU dataframe when comparing performance.  Writing to File cuDF also provides methods for writing data to files. Here we create a new dataframe specifically containing residents of Blackpool county and then write it to `blackpool.csv`, before doing the same with Pandas for comparison.  cuDF<jupyter_code>%time blackpool_residents = gdf.loc[gdf['county'] == 'Blackpool']
print(f'{blackpool_residents.shape[0]} residents')
%time blackpool_residents.to_csv('blackpool.csv')<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>%time blackpool_residents_pd = df.loc[df['county'] == 'Blackpool']
%time blackpool_residents_pd.to_csv('blackpool_pd.csv')<jupyter_output><empty_output><jupyter_text>Exercise: Initial Data Exploration Now that we have some data loaded, let's do some initial exploration.Use the `head`, `dtypes`, and `columns` methods on `gdf`, as well as the `value_counts` on individual `gdf` columns, to orient yourself to the data. If you're interested, use the `%time` magic command to compare performance against the same operations on the Pandas `df`. You can create additional interactive cells by clicking the `+` button above, or by switching to command mode with `Esc` and using the keyboard shortuts `a` (for new cell above) and `b` (for new cell below).If you fill up the GPU memory at any time, don't forget that you can restart the kernel and rerun the cells up to this point quite quickly.<jupyter_code># Begin your initial exploration here. Create more cells as needed.<jupyter_output><empty_output><jupyter_text>Basic Operations with cuDF Except for being much more performant with large datasets, cuDF looks and feels a lot like Pandas. In this section we highlight a few very simple operations. When performing data operations on cuDF dataframes, column operations are typically much more performant than row-wise operations.  Converting Data Types For machine learning later in this workshop, we will sometimes need to convert integer values into floats. Here we convert the `age` column from `int64` to `float32`, comparing performance with Pandas:  cuDF<jupyter_code>%time gdf['age'] = gdf['age'].astype('float32')<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>%time df['age'] = df['age'].astype('float32')<jupyter_output><empty_output><jupyter_text>Column-Wise Aggregations Similarly, column-wise aggregations take advantage of the GPU's architecture and RAPIDS' memory format.  cuDF<jupyter_code>%time gdf['age'].mean()<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>%time df['age'].mean()<jupyter_output><empty_output><jupyter_text>String Operations Although strings are not a datatype traditionally associated with GPUs, cuDF supports powerful accelerated string operations.  cuDF<jupyter_code>%time gdf['name'] = gdf['name'].str.title()
gdf.head()<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>%time df['name'] = df['name'].str.title()
df.head()<jupyter_output><empty_output><jupyter_text>Data Subsetting with `loc` and `iloc` cuDF also supports the core data subsetting tools `loc` (label-based locator) and `iloc` (integer-based locator).  Range Selection Our data's labels happen to be incrementing numbers, though as with Pandas, `loc` will include every value it is passed whereas `iloc` will give the half-open range (omitting the final value).<jupyter_code>gdf.loc[100:105]
gdf.iloc[100:105]<jupyter_output><empty_output><jupyter_text>`loc` with Boolean Selection We can use `loc` with boolean selections:  cuDF<jupyter_code># as of version 0.10, the startswith method returns a list, so we convert it back to a Series for efficiency
# in a future version, that method and other string methods will return a Series when appropriate
%time e_names = gdf.loc[cudf.Series(gdf['name'].str.startswith('E'))]
e_names.head()<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>%time e_names_pd = df.loc[df['name'].str.startswith('E')]<jupyter_output><empty_output><jupyter_text>Combining with NumPy Methods We can combine cuDF methods with NumPy methods. Here we use `np.logical_and` for elementwise boolean selection.  cuDF<jupyter_code>%time ed_names = gdf.loc[np.logical_and(gdf['name'].str.startswith('E'), gdf['name'].str.endswith('d'))]
ed_names.head()<jupyter_output><empty_output><jupyter_text>For better performance, we can use CuPy instead of NumPy, thereby performing the elementwise boolean `logical_and` operation on GPU.<jupyter_code>%time ed_names = gdf.loc[cudf.Series(cp.logical_and(cudf.Series(gdf['name'].str.startswith('E')), cudf.Series(gdf['name'].str.endswith('d'))))]
ed_names.head()<jupyter_output><empty_output><jupyter_text>Pandas<jupyter_code>%time ed_names_pd = df.loc[np.logical_and(df['name'].str.startswith('E'), df['name'].str.endswith('d'))]<jupyter_output><empty_output>