<jupyter_start><jupyter_text>NY State Freedom of Information Law Committee on Open Government Advisory Opinions Web Scraping Notebook The Committee on Open Government is responsible for overseeing the Freedom of Information Law in New York State by advising government agencies, the press and the the public. Their written advisory opinions are effectively compliance instructions and may serve as precedent in similar cases. According to Committee Chair, <NAME>,> Since its creation in 1974, more than24,000 written advisory opinions have been prepared by the Committee at the requestof government, the public and the news media. \[. . .\] Opinions prepared since early 1993 that have educational or precedential value aremaintained online, identified by means of a series of key phrases in separate indicescreated in relation to the Freedom of Information Law and the Open Meetings Law. However, that online access is largely unstructured and difficult to use. Opinions are given a single topic keyphrase, often using bureaucratic vocabulary, and grouped together on webpages by the first letter of their keyphrase. These pages, and the site in general, lack descriptive structures like titles, so the only available search feature (a google plugin) displays results that appear largely identical.To handcraft finding tools would be arduous and require the domain knowledge of the committee members, who do not themselves have experience in information storage and retrieval. As an alternative, text mining and machine learning methods can help to generate the data and structure needed for robust search and discovery features. This notebook is the first stage in that process: Data collection and initial cleaning.<jupyter_code>import pandas as pd
import numpy as np
from bs4 import BeautifulSoup
from requests import get
from requests.exceptions import RequestException
from contextlib import closing
import time
import pickle 
import re
#A GET url request function with error logging

def simple_get(url):

    try:
        with closing(get(url, stream=True)) as resp:
            if is_good_response(resp):
                return resp.content
            else:
                return None

    except RequestException as e:
        log_error('Error during requests to {0} : {1}'.format(url, str(e)))
        return None


def is_good_response(resp):

    content_type = resp.headers['Content-Type'].lower()
    return (resp.status_code == 200 
            and content_type is not None 
            and content_type.find('html') > -1)


def log_error(e):
    print(e)<jupyter_output><empty_output><jupyter_text>Gathering the Opinions---There are two layers we need to gather urls from. The first is the A-Z index, one page per letter. Then we travel to each of those pages and gather all of the links to individual opinions. The site salts the urls of the letters by periodically adding a different random letter before the letter of the index. This may be an effort to prevent web scraping or could be an unintended behavior while serving the content. Either way, the page doesn't have a robots.txt file, so we scrape as politely as we can. This scraping code looks at the index page and gets the URLs of each letter and stores them in a list.---<jupyter_code>base = 'https://www.dos.ny.gov/coog/foil_listing/'

index_page = simple_get(base + 'findex.html')

index_soup = BeautifulSoup(index_page, 'html.parser')

index = index_soup.find_all('table')

index[0].find_all('td')

index_urls = []

for i in index[0].find_all('a'):
    index_urls.append(base + i['href'])<jupyter_output><empty_output><jupyter_text>Getting the Opinions and Relevant Metata---We structure the data as a dictionary, with the Advisory Committee keyphrases as keys. We travel to each letter and capture all of the keyphrases and any associated links to opinions. We then travel any linked opinions and store the opinion text. We keep the html markup because it contains useful information, like paragraphing and typographic elements, that we can mine later. ---<jupyter_code>tag_dict = {}

#iterate through alphebetical index
for url in index_urls:
    
    #get the html from the index page
    index_page_html = simple_get(url)
    
    #pass over any dead links
    if index_page_html == False:
        pass
    
    else:

        #This html is particularyly soupy. We need to grab every other 'td',
        # table_side switches from positive to negative for left and right td's
        table_side = -1
        
        #parse the page 
        soup = BeautifulSoup(index_page_html, "html.parser")

        #grabs the second table
        table =  soup.find_all('table')[1]

        for item in table.find_all('td'):

            #grabs the text from the leftside td, which is keyword name
            if table_side == -1:
                text = item.text

            #makes a list of the links and the non-linked codes for rulings from the rightside td
            else:
                tag_dict[text] = [[a['href'] for a in item.find_all('a')], item.text.replace(',','').split()]

            #flips the sign on the count to grab from right/left
            table_side *= -1

        #to be polite to the server
        time.sleep((np.random.randint(100,200))/100)<jupyter_output><empty_output><jupyter_text>Store Opinions and IDs---Opinion IDs will help us distinguish the publish from the unpublished opinions later when we process the PDFs of additional opinions we recieved from the Committee.---<jupyter_code>#add a list for each key to hold the opinions
for data_list in tag_dict.values():
    data_list.append([])
    
    #if there were any urls for the keyword
    if len(data_list[key][0]) > 0:
    
        #iter through the urls and get the html of the ruling
        for url in data_list[0]:
            ruling = simple_get(url)

            #handle dead links
            if ruling == None:
                data_list[2].append('dead link')

            #only store the div with the content, the rest is formatting
            else:
                soup = BeautifulSoup(ruling, "html.parser")
                data_list[2].append(soup.find_all('div', id='mainContent')[0].prettify())

                #take between 1.0 and 1.5 seconds between calls to be polite to the server
                #(and avoid any potention )
                time.sleep((np.random.randint(100,150)/100))<jupyter_output><empty_output><jupyter_text>Cleaning---Here we do some basic cleaning to get the plain text from the html and store it. For this process to scale, we would simply convert this to a function and serve it directly to whichever model we were using. After cleaning, we pickle the dictionary and save it to the hard drive. ---<jupyter_code>#add a new list to hold the plain text opinion
for data_list in tag_dict.values():
    data_list.append([])
    for i in range(len(data_list[2])):
        
        #pass over dead links
        if data_list[2][i] == 'dead link':
            pass
        
        #clean up the text for bag of words (bow) approach
        else:

            #grab the text from the div
            goup = BeautifulSoup(data_list[2][i]).div

            #Remove the JavaScript
            goup = goup.get_text().replace('document.write("OML-AO-"+DocName);', '')

            #Remove the carriage return/newline because for bow it doesn't matter
            goup = re.sub(r'\s+',' ', goup )

            #replace escaped single quotes with double quotes
            goup = re.sub(r'\'', '"', goup)
            
            #add the cleaned text to the 
            data_list.append(goup)

with open('FOILAdvisoryDecisionDataWithText.pickle', 'wb') as f:
    # Pickle the dictionary using the highest protocol available.
    pickle.dump(tag_dict, f, pickle.HIGHEST_PROTOCOL)<jupyter_output><empty_output><jupyter_text>Confirm ----As a precaution, make sure the data saved correctly before closing the notebook.----<jupyter_code>with open('FOILAdvisoryDecisiondataWithText.pickle', 'rb') as f:
    # The protocol version used is detected automatically, so we do not
    # have to specify it.
    tester = pickle.load(f)
len(tester) == len(tag_dict)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup