# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: spark
#     language: python
#     name: spark
# ---

# # Spark

# ## Spark Concepts
#
# ### History
#
# - Motivation
#     - Move computing to data, not data to computing
# - Google
#     Google Distributed Filesystem (GFS)
#     Big Table
#     Map-reduce
# - Yahoo!
#     - Hadoop Distributed File System (HDFS)
#     - Yet Anohter Resource Negotiator (YARN)
#     - MapReduce
# - Limitations of MapReduce
#     - Cumbersome API
#     - Every stage reads from/writes to disk
#     - No native interactive SQL (HIVE, Impala, Drill)
#     - No native streaming (Storm)
#     - No native mahcine learning (Mahout)
# - Spark
#     - Simple API
#     - In-memory storage
#     - Better fault tolerance
#     - Can take advantage of embarrassingly parallel computations
#     - Multi-language support (Scala, Java, Python, R)
#     - Support multiple workloads
#     - Spark 1.0 released May 11, 2014
# - Speed
#     - DAG scheduler
#     - Catalyst query optimizer
#     - Tungsten execution engine
# - Ease of use
#     - Resilient Distributed Dataset (RDD)
#     - Lazy evaluation
# - Modularity
#     - Languages
#     - APIs - Strucrtured Srtreaming, GraphX, MLLib
# - Extensible
#     - Data readers
#     - Data writers
#     - [3rd party connectors](https://spark.apache.org/third-party-projects.html)
#     - Multiple platforms - local, data center, cloud, kubernetes
# - Distributed execution concepts
#     - Spark driver
#         - Spark session
#         - Spark shell
#         - Communicates with Spark Master
#         - Communicates with Spakr workers
#     - Spark master
#         - Resource management on cluster
#     - Spark workers
#         - Communicate reosuces to cluster manger
#         - Start Spark Executors
#     - Spark executors
#        - Communicate with driver
#        - Runs task
#        - Can run multiple threds in parallel
# - Execution process
#     - Driver creates jobs
#         - Each job is a DAG
#         - DAGScheduler translates into physical plan using RDDs
#         - Optimization incldues merging and splitiing into stages
#         - TaskScheduler distributes physical plans to Executors
#     - Job consists of one or more stages
#         - Stage normally ends when there is a need to exchange data (shuffle)
#     - Stage consists of tasks
#         - A task is a unit of execution
#         - Each task is sent to one executor and assigned one data partition
#         - A mutli-core computer can run several tasks in parallel
# - Transforms and actions
#     - Transforms are lazy
#     - Actions are eager
#     - NArrow versus broad transforms

# ### Resources
#
# - [Quick Start](http://spark.apache.org/docs/latest/quick-start.html)
# - [Spark Programming Guide](http://spark.apache.org/docs/latest/programming-guide.html)
# - [DataFramews, DataSets and SQL](http://spark.apache.org/docs/latest/sql-programming-guide.html)
# - [MLLib](http://spark.apache.org/docs/latest/mllib-guide.html)
# - [GraphX](http://spark.apache.org/docs/latest/graphx-programming-guide.html)
# - [Streaming](http://spark.apache.org/docs/latest/streaming-programming-guide.html)

# ## Distributed computing bakkground
#
# With distributed computing, you interact with a network of computers that communicate via message passing as if issuing instructions to a single computer.
#
# ![Distributed computing](https://image.slidesharecdn.com/distributedcomputingwithspark-150414042905-conversion-gate01/95/distributed-computing-with-spark-21-638.jpg?)
#
# Source: https://image.slidesharecdn.com/distributedcomputingwithspark-150414042905-conversion-gate01/95/distributed-computing-with-spark-21-638.jpg
#
# ### Hadoop and Spark
#
# - There are 3 major components to a distributed system
#     - storage
#     - cluster management
#     - computing engine
#
# - Hadoop is a framework that provides all 3 
#     - distributed storage (HDFS) 
#     - clsuter managemnet (YARN)
#     - computing eneine (MapReduce)
#     
# - Spakr only provides the (in-memory) distributed computing engine, and relies on other frameworks for storage and cluster management. It is most frequently used on top of the Hadoop framework, but can also use other distributed storage(e.g. S3 and Cassandra) or cluster mangement (e.g. Mesos) software.
#
# ### Distributed storage
#
# ![storage](http://slideplayer.com/slide/3406872/12/images/15/HDFS+Framework+Key+features+of+HDFS:.jpg)
#
# Source: http://slideplayer.com/slide/3406872/12/images/15/HDFS+Framework+Key+features+of+HDFS:.jpg
#
# ### Role of YARN
#
# - Resource manager (manages cluster resources)
#     - Scheduler
#     - Applications manager
# - Node manager (manages single machine/node)
#     - manages data containers/partitions
#     - monitors resource usage
#     - reports to resource manager
#
# ![Yarn](https://kannandreams.files.wordpress.com/2013/11/yarn1.png)
#
# Source: https://kannandreams.files.wordpress.com/2013/11/yarn1.png
#
# ### YARN operations
#
# ![yarn ops](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/yarn_architecture.gif)
#
# Source: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/yarn_architecture.gif
#
# ### Hadoop MapReduce versus Spark
#
# Spark has several advantages over Hadoop MapReduce
#
# - Use of RAM rahter than disk mean faster processing for multi-step operations
# - Allows interactive applications
# - Allows real-time applications
# - More flexible programming API (full range of functional constructs)
#
# ![Hadoop](https://i0.wp.com/s3.amazonaws.com/acadgildsite/wordpress_images/bigdatadeveloper/10+steps+to+master+apache+spark/hadoop_spark_1.png)
#
# Source: https://i0.wp.com/s3.amazonaws.com/acadgildsite/wordpress_images/bigdatadeveloper/10+steps+to+master+apache+spark/hadoop_spark_1.png
#
# ### Overall Ecosystem
#
# ![spark](https://cdn-images-1.medium.com/max/1165/1*z0Vm749Pu6mHdlyPsznMRg.png)
#
# Source: https://cdn-images-1.medium.com/max/1165/1*z0Vm749Pu6mHdlyPsznMRg.png
#
# ### Spark Ecosystem
#
# - Spark is written in Scala, a functional programming language built on top of the Java Virtual Machine (JVM)
# - Traditionally, you have to code in Scala to get the best performance from Spark
# - With Spark DataFrames and vectorized operations (Spark 2.3 onwards) Python is now competitive
#
# ![eco](https://databricks.com/wp-content/uploads/2018/12/Spark.jpg)
#
# Source: https://databricks.com/wp-content/uploads/2018/12/Spark.jpg
#
# ### Livy and Spark magic
#
# - Livy provides a REST interface to a Spark cluster.
#
# ![Livy](https://cdn-images-1.medium.com/max/956/0*-lwKpnEq0Tpi3Tlj.png)
#
# Source: https://cdn-images-1.medium.com/max/956/0*-lwKpnEq0Tpi3Tlj.png
#
# ### PySpark
#
# ![PySpark](http://i.imgur.com/YlI8AqEl.png)
#
# Source: http://i.imgur.com/YlI8AqEl.png
#
# ### Resilient distributed datasets (RDDs)
#
# ![rdd](https://mapr.com/blog/real-time-streaming-data-pipelines-apache-apis-kafka-spark-streaming-and-hbase/assets/blogimages/msspark/imag12.png)
#
# Source: https://mapr.com/blog/real-time-streaming-data-pipelines-apache-apis-kafka-spark-streaming-and-hbase/assets/blogimages/msspark/imag12.png
#
# ### Spark fault tolerance
#
# ![graph](https://image.slidesharecdn.com/deep-dive-with-spark-streamingtathagata-dasspark-meetup2013-06-17-130623151510-phpapp02/95/deep-dive-with-spark-streaming-tathagata-das-spark-meetup-20130617-13-638.jpg)
#
# Source: https://image.slidesharecdn.com/deep-dive-with-spark-streamingtathagata-dasspark-meetup2013-06-17-130623151510-phpapp02/95/deep-dive-with-spark-streaming-tathagata-das-spark-meetup-20130617-13-638.jpg

# ## Install Spark
#
# - If necessary, install [Java](https://java.com/en/download/help/download_options.xml)
# - Downlaod and install [Sppark](http://spark.apache.org/downloads.html)
#
# ```bash
# wget http://apache.mirrors.pair.com/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
# tar xzf spark-2.4.4-bin-hadoop2.7.tgz
# sudo mv spark-2.4.4-bin-hadoop2.7 /usr/local/spark
# ```
#
# Set up graphframes
#
# ```bash
# pip install graphframes
# ```
#
# Set up environment variables
#
# ```bash
# export PATH=$PATH:/usr/local/spark/bin
# export SPARK_HOME=/usr/local/spark
# export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
# export PYSPARK_DRIVER_PYTHON="jupyter"
# export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
# export PYSPARK_PYTHON=python3
# ```

# ### If you want `sparkmagic`
#
# Install and start `livy`
# ```
# # cd ~
# wget http://apache.osuosl.org/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
# unzip apache-livy-0.6.0-incubating-bin.zip
# # mv apache-livy-0.6.0-incubating-bin livy
# livy/bin/livy-server start
# ```
#
# Install `sparkmagic`
#
# ```
# pip install sparkmagic
# jupyter nbextension enable --py --sys-prefix widgetsnbextension 
# ```
#
# Type `pip show sparkmagic` and cd to the directory shown in LOCATION
#
# ```
# jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
# jupyter serverextension enable --py sparkmagic
# ```
#
# For the adventurous, see [Running Spark on an AWS EMR cluster](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark.html).

# ## Check

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# ### Spark UI
#
# - Default port 4040 http://localhost:4040/

# %%file candy.csv
name,age,candy
tom,3,m&m
shirley,6,mentos
david,4,candy floss
anne,5,starburst

df = spark.read.csv('candy.csv')

df.show(n=10)

spark.stop()



StarcoderdataJupyterScriptsDedupFiltered