<jupyter_start><jupyter_text>Step 1: Import the requests library<jupyter_code>import requests<jupyter_output><empty_output><jupyter_text>Step 2: Send an HTTP request, get the response, and save in a variable<jupyter_code>response = requests.get("http://www.epicurious.com/search/Tofu+Chili")<jupyter_output><empty_output><jupyter_text>Step 3: Check the response status code to see if everything went as plannedstatus code 200: the request response cycle was successfulany other status code: it didn't work (e.g., 404 = page not found)<jupyter_code>print(response.status_code)<jupyter_output><empty_output><jupyter_text>Step 4: Get the content of the responseConvert to utf-8 if necessary<jupyter_code>response.content.decode('utf-8')<jupyter_output><empty_output><jupyter_text>Problem: Get the contents of Wikipedia's main page and look for the string "Did you know" in it<jupyter_code>url = "https://en.wikipedia.org/wiki/main_page"
#The rest of your code should go below this line<jupyter_output><empty_output><jupyter_text>JSONThe python library - json - deals with converting text to and from JSON<jupyter_code>import json
data_string = '[{"b": [2, 4], "c": 3.0, "a": "A"}]'
python_data = json.loads(data_string)
print(python_data)<jupyter_output><empty_output><jupyter_text>json.loads recursively decodes a string in JSON format into equivalent python objectsdata_string's outermost element is converted into a python listthe first element of that list is converted into a dictionarythe key of that dictionary is converted into a stringthe value of that dictionary is converted into a list of two integer elements<jupyter_code>print(type(data_string),type(python_data))
print(type(python_data[0]),python_data[0])
print(type(python_data[0]['b']),python_data[0]['b'])<jupyter_output><empty_output><jupyter_text>json.loads will throw an exception if the format is incorrect<jupyter_code>#Wrong
# json.loads("Hello")
#Correct
json.loads('"Hello"')
import json
data_string = json.dumps(python_data)
print(type(data_string))
print(data_string)<jupyter_output><empty_output><jupyter_text>requests library and JSON<jupyter_code>address="Columbia University, New York, NY"
url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
response = requests.get(url).json()
print(type(response))<jupyter_output><empty_output><jupyter_text>Exception checking!<jupyter_code>address="Columbia University, New York, NY"
url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
try:
    response = requests.get(url)
    if not response.status_code == 200:
        print("HTTP error",response.status_code)
    else:
        try:
            response_data = response.json()
        except:
            print("Response not in valid JSON format")
except:
    print("Something went wrong with requests.get")
print(response_data)
# def id_generator(dict_var, search):
#       for k, v in dict_var.items():
#             if k == search:
#                  yield v
#             elif isinstance(v, dict):
#                  for id_val in id_generator(v):
#                        yield id_val
# 
# for _ in id_generator(response_data, 'long_name'):
#      print(_)
# The JSON might contain a list of objects, which needs to be searched:
def item_generator1(json_input, lookup_key):
    if isinstance(json_input, dict):
        for k, v in json_input.items():
            if k == lookup_key:
                yield v
            else:
                for child_val in item_generator(v, lookup_key):
                    yield child_val
    elif isinstance(json_input, list):
        for item in json_input:
            for item_val in item_generator(item, lookup_key):
                yield item_val
# The JSON might contain a list of objects, which needs to be searched:
def item_generator(json_input, lookup_key):
    if isinstance(json_input, dict):
        for k, v in json_input.items():
            if k == lookup_key:
                yield {k:v}
            else:
                for child_val in item_generator(v, lookup_key):
                    yield child_val
    elif isinstance(json_input, list):
        for item in json_input:
            for item_val in item_generator(item, lookup_key):
                yield item_val

def get_json_dict(json_data, lookup_key):
    return list(item_generator(json_data, lookup_key))
print(get_json_dict(response_data, 'location'))
for x in get_json_dict(response_data, 'location'):
    print(x)
for v in item_generator(response_data, 'address_components'):
     print(v)
item_generator(response_data, 'address_components')<jupyter_output><empty_output><jupyter_text>Problem 1: Write a function that takes an address as an argument and returns a (latitude, longitude) tuple<jupyter_code>def get_lat_lng(address_string):
    #python code goes here<jupyter_output><empty_output><jupyter_text>Problem 2: Extend the function so that it takes a possibly incomplete address as an argument and returns a list of tuples of the form (complete address, latitude, longitude)<jupyter_code>def get_lat_lng(address_string):
    #python code goes here<jupyter_output><empty_output><jupyter_text>XMLThe python library - lxml - deals with converting an xml string to python objects and vice versa<jupyter_code>data_string = """
<Bookstore>
   <Book ISBN="ISBN-13:978-1599620787" Price="15.23" Weight="1.5">
      <Title>New York Deco</Title>
      <Authors>
         <Author Residence="New York City">
            <First_Name>Richard</First_Name>
            <Last_Name>Berenholtz</Last_Name>
         </Author>
      </Authors>
   </Book>
   <Book ISBN="ISBN-13:978-1579128562" Price="15.80">
      <Remark>
      Five Hundred Buildings of New York and over one million other books are available for Amazon Kindle.
      </Remark>
      <Title>Five Hundred Buildings of New York</Title>
      <Authors>
         <Author Residence="Beijing">
            <First_Name>Bill</First_Name>
            <Last_Name>Harris</Last_Name>
         </Author>
         <Author Residence="New York City">
            <First_Name>Jorg</First_Name>
            <Last_Name>Brockmann</Last_Name>
         </Author>
      </Authors>
   </Book>
</Bookstore>
"""
from lxml import etree
root = etree.XML(data_string)
print(root.tag,type(root.tag))
print(etree.tostring(root, pretty_print=True).decode("utf-8"))<jupyter_output><empty_output><jupyter_text>Iterating over an XML treeUse an iterator. The iterator will generate every tree element for a given subtree<jupyter_code>for element in root.iter():
    print(element)<jupyter_output><empty_output><jupyter_text>Or just use the child in subtree construction<jupyter_code>for child in root:
    print(child)<jupyter_output><empty_output><jupyter_text>Accessing the tag<jupyter_code>for child in root:
    print(child.tag)<jupyter_output><empty_output><jupyter_text>Using the iterator to get specific tagsIn the below example, only the author tags are accessedFor each author tag, the .find function accesses the First_Name and Last_Name tagsThe .find function only looks at the children, not other descendants, so be careful!The .text attribute prints the text in a leaf node<jupyter_code>for element in root.iter("Author"):
    print(element.find("Last_Name").text, element.find("First_Name").text)<jupyter_output><empty_output><jupyter_text>Problem: Find the last names of all authors in the tree “root” using xpath<jupyter_code>for element in root.findall('Book/Authors/Author/Last_Name'):
    print(element.text)
for element in root.findall('Book/Title'):
    print(element.text)<jupyter_output><empty_output><jupyter_text>Using values of attributes as filtersExample: Find the first name of the author of a book that weighs 1.5 oz<jupyter_code>root.find('Book[@Weight="1.5"]/Authors/Author/First_Name').text<jupyter_output><empty_output><jupyter_text>Problem: Print first and last names of all authors who live in New York City<jupyter_code>root.find('Book/Authors/Author[@<EMAIL>="New <EMAIL>"]/Last_Name').text
for element in root.findall('Book/Authors/Author[@<EMAIL>="New <EMAIL>"]'):
    print(element.find("Last_Name").text, element.find("First_Name").text)<jupyter_output><empty_output>