# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Plotting massive data sets
#
# **This notebook is experimental in pydeck's beta version. It may not work on all devices.**
#
# This notebook plots 1.6 million points of LIDAR points around the Carnegie Mellon University campus. ([Source](https://github.com/ajduberstein/oakland_point_cloud)) The data points are labeled. With pydeck, we can render these points and interact with them. We'll downsample this data to 800,000 points–depending on your device, you may be able to run the full data set.

# ### Cleaning the data
#
# First we need to import the data. We should expect about 1.6M points.

# +
import pandas as pd

URL = 'https://raw.githubusercontent.com/ajduberstein/oakland_point_cloud/master/%s'
DATA_URL_1 = URL % 'lidar_chunks_1.csv'
DATA_URL_2 = URL % 'lidar_chunks_2.csv'
LOOKUP_URL = URL % 'ground_truth_label.csv'
lidar = pd.concat([pd.read_csv(DATA_URL_1), pd.read_csv(DATA_URL_2)])
lookup = pd.read_csv(LOOKUP_URL)
lidar = lidar.merge(lookup)
# -

print('Number of points:', lidar.count()[0])

# It does not appear to be in a standard coordinate format, so we'll scale it to make it easy to plot on a map. We'll also color objects by label type. The `data_utils.assign_random_colors` assigns a random RGB value to a vector of data labels.

# +
from pydeck import data_utils

color_lookup = data_utils.assign_random_colors(lidar['label_name'])
lidar['rgb'] = lidar.apply(lambda row: color_lookup.get(row['label_name']), axis=1)
# Scaling the points using min-max scaling
lidar[['x', 'y', 'z']] -= lidar[['x', 'y', 'z']].max()
lidar[['x', 'y', 'z']] /= lidar[['x', 'y', 'z']].min()
lidar[['x', 'y']] /= 1000
lidar[['z']] *= 10
lidar.head()
# -

# ### Plotting the data
#
# We'll define a single `PointCloudLayer` and plot it.
#
# pydeck by default expects the input of `get_position` to be a string name indicating a single position value. For convenience, you can pass in a string indicating the X/Y/Z coordinate, here `get_position='[x, y, z]'`.
#
# We'll zoom to the approximate center of the data by taking a mean of a few hundred points in pandas.
#
# This example may take 10-15 seconds to render.

# +
import pydeck as pdk

point_cloud = pdk.Layer(
    'PointCloudLayer',
    lidar[['x', 'y', 'z', 'label_name', 'rgb']].sample(800000),
    # You can specify the XYZ coordinate in a list as a string
    get_position='[x, y, -1*z]',
    get_normal=[0, 0, 1],
    get_color='rgb',
    pickable=True,
    auto_highlight=True,
    radius_pixels=3)


view_state = pdk.data_utils.compute_view(lidar[['x', 'y']])
view_state.max_zoom = 23
view_state.zoom = 22
view_state.pitch = 60


r = pdk.Deck(
    point_cloud,
    initial_view_state=view_state,
    map_style='',
    width=700)
r.show()

# + language="javascript"
# // Use Javascript to change the cell background to black
# document.getElementById("deckgl-overlay").style.backgroundColor = "black"
# -

# #### Citations:
#
# Contextual Classification with Functional Max-Margin Markov Networks. 
# <NAME>, <NAME> (Drew) Bagnell, <NAME>, and <NAME>. 
# IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June, 2009.

StarcoderdataJupyterScriptsDedupFiltered