<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)  Use pretrained `explain_document` Pipeline  Stages * DocumentAssembler * SentenceDetector * Tokenizer * Lemmatizer * Stemmer * Part of Speech * SpellChecker (Norvig)<jupyter_code>import sys
import time

#Spark ML and SQL
from pyspark.ml import Pipeline, PipelineModel
from pyspark.sql.functions import array_contains
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
#Spark NLP
import sparknlp
from sparknlp.pretrained import PretrainedPipeline
from sparknlp.annotator import *
from sparknlp.common import RegexRule
from sparknlp.base import DocumentAssembler, Finisher<jupyter_output><empty_output><jupyter_text>Let's create a Spark Session for our app<jupyter_code>spark = sparknlp.start()

print("Spark NLP version: ", sparknlp.version())
print("Apache Spark version: ", spark.version)<jupyter_output>Spark NLP version:  2.6.0
Apache Spark version:  2.4.4<jupyter_text>This is our testing document, we'll use it to exemplify all different pipeline stages.<jupyter_code>testDoc = [
"Frenchg author who helped pioner the science-fiction genre. \
Verne wrate about space, aisr, and underwater travel befdaore \
navigable aircrast and practical submarines were invented, \
and before any means of space travel had been devised. "
]
pipeline = PretrainedPipeline('explain_document_ml', lang='en')<jupyter_output>explain_document_ml download started this may take some time.
Approx size to download 9.4 MB
[OK!]<jupyter_text>We are not interested in handling big datasets, let's switch to LightPipelines for speed.<jupyter_code>result = pipeline.annotate(testDoc)<jupyter_output><empty_output><jupyter_text>Let's analyze these results - first let's see what sentences we detected<jupyter_code>[content['sentence'] for content in result]<jupyter_output><empty_output><jupyter_text>Now let's see how those sentences were tokenized<jupyter_code>[content['token'] for content in result]<jupyter_output><empty_output><jupyter_text>Notice some spelling errors? the pipeline takes care of that as well<jupyter_code>[content['spell'] for content in result]<jupyter_output><empty_output><jupyter_text>Now let's see the lemmas<jupyter_code>[content['lemmas'] for content in result]<jupyter_output><empty_output><jupyter_text>Let's check the stems, any difference with the lemmas shown bebore?[content['lemmas'] for content in result]<jupyter_code>[content['stems'] for content in result]<jupyter_output><empty_output><jupyter_text>Now it's the turn on Part Of Speech(POS)<jupyter_code>pos = [content['pos'] for content in result]
token = [content['token'] for content in result]
# let's put token and tag together
list(zip(token[0], pos[0]))<jupyter_output><empty_output><jupyter_text>Use pretrained `match_chunk` Pipeline for Individual Noun Phrase  * DocumentAssembler* SentenceDetector* Tokenizer* Part of speech* chunkerPipeline:* The pipeline uses regex `?*+`* which states that whenever the chunk finds an optional determiner * (DT) followed by any number of adjectives (JJ) and then a noun (NN) then the Noun Phrase(NP) chunk should be formed.<jupyter_code>pipeline = PretrainedPipeline('match_chunks', lang='en')
result = pipeline.annotate("The book has many chapters") # single noun phrase
result['chunk']
result = pipeline.annotate("the little yellow dog barked at the cat") #multiple noune phrases
result['chunk']
result<jupyter_output><empty_output>