We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
OH failing since at least 2018-04-18
OH has been failing since 2018-04-18

Based on automated runs it appears that OH has not run successfully in 2 days (2018-04-18).


```
  02:46:44 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb602/disapprove
02:46:45 INFO pupa: save bill HB 602 in 132 as bill_cb814bda-43a5-11e8-a12a-020dd716e63e.json
02:46:45 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/
02:46:46 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/actions
02:46:47 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/votes
02:46:48 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/cmtevotes
02:46:49 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/veto
02:46:50 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/disapprove
02:46:51 INFO pupa: save bill HB 603 in 132 as bill_cf14d0f0-43a5-11e8-a12a-020dd716e63e.json
02:46:51 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/
02:46:52 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/actions
02:46:53 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/votes
02:46:54 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/cmtevotes
02:46:55 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/veto
02:46:56 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/disapprove
02:46:57 INFO pupa: save bill HB 604 in 132 as bill_d2b4f550-43a5-11e8-a12a-020dd716e63e.json
02:46:57 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/
02:46:58 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/actions
02:46:59 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/votes
02:47:00 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/cmtevotes
02:47:01 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/veto
02:47:02 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/disapprove
02:47:03 INFO pupa: save bill HB 605 in 132 as bill_d6465452-43a5-11e8-a12a-020dd716e63e.json
02:47:03 INFO scrapelib: GET - http://www.ohiosenate.gov/members/senate-directory
loaded Open States pupa settings...
oh (scrape, import)
  bills: {}
  people: {}
Traceback (most recent call last):
  File "/opt/openstates/venv-pupa//bin/pupa", line 11, in <module>
    load_entry_point('pupa', 'console_scripts', 'pupa')()
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py", line 68, in main
    subcommands[args.subcommand].handle(args, other)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 260, in handle
    return self.do_handle(args, other, juris)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 305, in do_handle
    report['scrape'] = self.do_scrape(juris, args, scrapers)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py", line 173, in do_scrape
    report[scraper_name] = scraper.do_scrape(**scrape_args)
  File "/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py", line 111, in do_scrape
    for obj in self.scrape(**kwargs) or []:
  File "/opt/openstates/openstates/openstates/oh/people.py", line 48, in scrape
    yield from self.scrape_page('upper', CHAMBER_URLS['upper'])
  File "/opt/openstates/openstates/openstates/oh/people.py", line 131, in scrape_page
    page = self.get(url).text
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/openstates/venv-pupa/lib/python3.5/site-packages/scrapelib/__init__.py", line 292, in request
    raise HTTPError(resp)
scrapelib.HTTPError: 404 while retrieving http://www.ohiosenate.gov/members/senate-directory
```

Visit http://bobsled.openstates.org for more info.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `openstates/oh/people.py`
Content:
```
1 import re
2 
3 from pupa.scrape import Person, Scraper, Organization
4 
5 import lxml.html
6 
7 
8 JOINT_COMMITTEE_OVERRIDE = [  # without Joint" in the name.
9     "State Controlling Board",
10     "Legislative Service Commission",
11     "Correctional Institution Inspection Committee"
12 ]
13 
14 SUBCOMMITTEES = {
15     # The Senate only has Finance subcommittees
16     "Finance - Corrections Subcommittee": "Finance",
17     "Finance - Education Subcommittee": "Finance",
18     "Finance - General Government Subcommittee": "Finance",
19     "Finance - Higher Ed Subcommittee": "Finance",
20     "Finance - Workforce Subcommittee": "Finance",
21 
22     # The House has mostly Finance, but also one more
23     "Community and Family Advancement Subcommittee on Minority Affairs":
24         "Community and Family Advancement",
25     "Finance Subcommittee on Agriculture Development and Natural Resources":
26         "Finance",
27     "Finance Subcommittee on Health and Human Services": "Finance",
28     "Finance Subcommittee on Higher Education": "Finance",
29     "Finance Subcommittee on Primary and Secondary Education": "Finance",
30     "Finance Subcommittee on Transportation": "Finance",
31     "Finance Subcommittee on State Government and Agency Review": "Finance"
32 }
33 
34 CHAMBER_URLS = {
35     'upper': "http://www.ohiosenate.gov/members/senate-directory",
36     'lower':  "http://www.ohiohouse.gov/members/member-directory"
37 }
38 
39 committee_cache = {}
40 
41 
42 class OHLegislatorScraper(Scraper):
43     def scrape(self, chamber=None):
44         self.committees = {}
45         if chamber:
46             yield from self.scrape_page(chamber, CHAMBER_URLS[chamber])
47         else:
48             yield from self.scrape_page('upper', CHAMBER_URLS['upper'])
49             yield from self.scrape_page('lower', CHAMBER_URLS['lower'])
50         yield from self.committees.values()
51 
52     def fetch_committee_positions(self, a):
53         page = self.get(a.attrib['href']).text
54         page = lxml.html.fromstring(page)
55         page.make_links_absolute(a.attrib['href'])
56         ret = {}
57         for entry in page.xpath("//div[@class='committeeMembers']//td//a"):
58             person = re.sub(
59                 "\s+", " ", re.sub("\(.*\)", "", entry.text or "")).strip()
60 
61             if person == "":
62                 continue
63 
64             title = entry.xpath(".//div[@class='title']/text()") or None
65 
66             if title:
67                 title = title[0]
68                 ret[person] = title
69 
70         return ret
71 
72     def scrape_homepage(self, leg, chamber, homepage):
73         page = self.get(homepage).text
74         page = lxml.html.fromstring(page)
75         page.make_links_absolute(homepage)
76         bio = page.xpath(
77             "//div[@class='biography']//div[@class='right']//p/text()")
78         if bio != []:
79             bio = bio[0]
80             leg.extras['biography'] = bio
81 
82         fax_line = [
83             x.strip() for x in
84             page.xpath(
85                 "//div[@class='contactModule']/div[@class='data']/text()"
86             )
87             if "Fax" in x
88         ]
89         if fax_line:
90             fax_number = re.search(
91                 r'(\(\d{3}\)\s\d{3}\-\d{4})', fax_line[0]
92             ).group(1)
93             leg.add_contact_detail(type='fax', value=fax_number, note='Capitol Office')
94 
95         ctties = page.xpath("//div[@class='committeeList']//a")
96         for a in ctties:
97             entry = a.text_content()
98 
99             if entry in committee_cache:
100                 committee_positions = committee_cache[entry]
101             else:
102                 committee_positions = self.fetch_committee_positions(a)
103                 committee_cache[entry] = committee_positions
104 
105             chmbr = "legislature" if "joint" in entry.lower() else chamber
106             if entry in JOINT_COMMITTEE_OVERRIDE:
107                 chmbr = "legislature"
108 
109             kwargs = {}
110 
111             if "subcommittee" in entry.lower():
112                 if entry in SUBCOMMITTEES:
113                     kwargs['subcommittee'] = entry
114                     entry = SUBCOMMITTEES[entry]
115                 else:
116                     self.warning("No subcommittee known: '%s'" % (entry))
117                     raise Exception
118             if (chmbr, entry) not in self.committees:
119                 org = Organization(
120                     name=entry,
121                     chamber=chmbr,
122                     classification='committee',
123                 )
124                 self.committees[(chmbr, entry)] = org
125             else:
126                 org = self.committees[(chmbr, entry)]
127             org.add_source(homepage)
128             leg.add_membership(org)
129 
130     def scrape_page(self, chamber, url):
131         page = self.get(url).text
132         page = lxml.html.fromstring(page)
133         page.make_links_absolute(url)
134 
135         for legislator in page.xpath(
136                 "//div[contains(concat(' ', normalize-space(@class), ' '), "
137                 "' memberModule ')]"
138                 ):
139             img = legislator.xpath(
140                 ".//div[@class='thumbnail']//img")[0].attrib['src']
141             data = legislator.xpath(".//div[@class='data']")[0]
142             homepage = data.xpath(".//a[@class='black']")[0]
143             full_name = homepage.text_content()
144 
145             if "Vacant" in full_name:
146                 continue
147 
148             homepage = homepage.attrib['href']
149             party = data.xpath(
150                 ".//span[@class='partyLetter']")[0].text_content()
151             party = {"R": "Republican", "D": "Democratic"}[party]
152             office_lines = data.xpath("child::text()")
153             phone = office_lines.pop(-1)
154             office = "\n".join(office_lines)
155             h3 = data.xpath("./h3")
156             if len(h3):
157                 h3 = h3[0]
158                 district = h3.xpath("./br")[0].tail.replace("District", ""
159                                                             ).strip()
160             else:
161                 district = re.findall(
162                     "\d+\.png",
163                     legislator.attrib['style']
164                 )[-1].split(".", 1)[0]
165 
166             full_name = re.sub("\s+", " ", full_name).strip()
167             email = (
168                 'rep{0:0{width}}@ohiohouse.gov'
169                 if chamber == 'lower' else
170                 'sd{0:0{width}}@ohiosenate.gov'
171             ).format(int(district), width=2)
172 
173             leg = Person(name=full_name, district=district,
174                          party=party, primary_org=chamber,
175                          image=img)
176 
177             leg.add_contact_detail(type='address', value=office, note='Capitol Office')
178             leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')
179             leg.add_contact_detail(type='email', value=email, note='Capitol Office')
180 
181             self.scrape_homepage(leg, chamber, homepage)
182 
183             leg.add_source(url)
184             leg.add_link(homepage)
185             yield leg
186
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/openstates/oh/people.py b/openstates/oh/people.py
--- a/openstates/oh/people.py
+++ b/openstates/oh/people.py
@@ -32,7 +32,7 @@
 }
 
 CHAMBER_URLS = {
-    'upper': "http://www.ohiosenate.gov/members/senate-directory",
+    'upper': "http://www.ohiosenate.gov/senators",
     'lower':  "http://www.ohiohouse.gov/members/member-directory"
 }
 
@@ -45,8 +45,8 @@
         if chamber:
             yield from self.scrape_page(chamber, CHAMBER_URLS[chamber])
         else:
-            yield from self.scrape_page('upper', CHAMBER_URLS['upper'])
-            yield from self.scrape_page('lower', CHAMBER_URLS['lower'])
+            yield from self.scrape_senator_page('upper', CHAMBER_URLS['upper'])
+            yield from self.scrape_member_page('lower', CHAMBER_URLS['lower'])
         yield from self.committees.values()
 
     def fetch_committee_positions(self, a):
@@ -127,7 +127,7 @@
             org.add_source(homepage)
             leg.add_membership(org)
 
-    def scrape_page(self, chamber, url):
+    def scrape_member_page(self, chamber, url):
         page = self.get(url).text
         page = lxml.html.fromstring(page)
         page.make_links_absolute(url)
@@ -183,3 +183,44 @@
             leg.add_source(url)
             leg.add_link(homepage)
             yield leg
+
+    def scrape_senator_page(self, chamber, url):
+        page = self.get(url).text
+        page = lxml.html.fromstring(page)
+        page.make_links_absolute(url)
+
+        for legislator in page.xpath("//div[contains(concat(' ', normalize-space(@class), ' '), "
+                                     "' portraitContainer ')]"):
+            img = legislator.xpath(".//div[@class='profileThumbnailBoundingBox']/@style")[0]
+            img = img[img.find('(')+1:img.find(')')]
+            full_name = legislator.xpath(".//div[@class='profileName']/a/text()")[0]
+            homepage_url = legislator.xpath(".//a[@class='profileImageLink']")[0].attrib['href']
+            district = legislator.xpath(".//div[@class='profileDistrict']"
+                                        "/a/text()")[0].split("#")[1]
+
+            if "Vacant" in full_name:
+                continue
+
+            homepage = self.get(homepage_url).text
+            page = lxml.html.fromstring(homepage)
+            phone = page.xpath("//div[@class='phone']/span/text()")[0]
+
+            address_lines = page.xpath("//div[@class='address']/span/text()")
+            address = "\n".join(address_lines)
+
+            email = (
+                'rep{0:0{width}}@ohiohouse.gov'
+                if chamber == 'lower' else
+                'sd{0:0{width}}@ohiosenate.gov'
+            ).format(int(district), width=2)
+
+            leg = Person(name=full_name, district=district,
+                         primary_org=chamber, image=img)
+
+            leg.add_contact_detail(type='address', value=address, note='Capitol Office')
+            leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')
+            leg.add_contact_detail(type='email', value=email, note='Capitol Office')
+
+            leg.add_source(url)
+            leg.add_link(homepage_url)
+            yield leg

{"golden_diff": "diff --git a/openstates/oh/people.py b/openstates/oh/people.py\n--- a/openstates/oh/people.py\n+++ b/openstates/oh/people.py\n@@ -32,7 +32,7 @@\n }\n \n CHAMBER_URLS = {\n-    'upper': \"http://www.ohiosenate.gov/members/senate-directory\",\n+    'upper': \"http://www.ohiosenate.gov/senators\",\n     'lower':  \"http://www.ohiohouse.gov/members/member-directory\"\n }\n \n@@ -45,8 +45,8 @@\n         if chamber:\n             yield from self.scrape_page(chamber, CHAMBER_URLS[chamber])\n         else:\n-            yield from self.scrape_page('upper', CHAMBER_URLS['upper'])\n-            yield from self.scrape_page('lower', CHAMBER_URLS['lower'])\n+            yield from self.scrape_senator_page('upper', CHAMBER_URLS['upper'])\n+            yield from self.scrape_member_page('lower', CHAMBER_URLS['lower'])\n         yield from self.committees.values()\n \n     def fetch_committee_positions(self, a):\n@@ -127,7 +127,7 @@\n             org.add_source(homepage)\n             leg.add_membership(org)\n \n-    def scrape_page(self, chamber, url):\n+    def scrape_member_page(self, chamber, url):\n         page = self.get(url).text\n         page = lxml.html.fromstring(page)\n         page.make_links_absolute(url)\n@@ -183,3 +183,44 @@\n             leg.add_source(url)\n             leg.add_link(homepage)\n             yield leg\n+\n+    def scrape_senator_page(self, chamber, url):\n+        page = self.get(url).text\n+        page = lxml.html.fromstring(page)\n+        page.make_links_absolute(url)\n+\n+        for legislator in page.xpath(\"//div[contains(concat(' ', normalize-space(@class), ' '), \"\n+                                     \"' portraitContainer ')]\"):\n+            img = legislator.xpath(\".//div[@class='profileThumbnailBoundingBox']/@style\")[0]\n+            img = img[img.find('(')+1:img.find(')')]\n+            full_name = legislator.xpath(\".//div[@class='profileName']/a/text()\")[0]\n+            homepage_url = legislator.xpath(\".//a[@class='profileImageLink']\")[0].attrib['href']\n+            district = legislator.xpath(\".//div[@class='profileDistrict']\"\n+                                        \"/a/text()\")[0].split(\"#\")[1]\n+\n+            if \"Vacant\" in full_name:\n+                continue\n+\n+            homepage = self.get(homepage_url).text\n+            page = lxml.html.fromstring(homepage)\n+            phone = page.xpath(\"//div[@class='phone']/span/text()\")[0]\n+\n+            address_lines = page.xpath(\"//div[@class='address']/span/text()\")\n+            address = \"\\n\".join(address_lines)\n+\n+            email = (\n+                'rep{0:0{width}}@ohiohouse.gov'\n+                if chamber == 'lower' else\n+                'sd{0:0{width}}@ohiosenate.gov'\n+            ).format(int(district), width=2)\n+\n+            leg = Person(name=full_name, district=district,\n+                         primary_org=chamber, image=img)\n+\n+            leg.add_contact_detail(type='address', value=address, note='Capitol Office')\n+            leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')\n+            leg.add_contact_detail(type='email', value=email, note='Capitol Office')\n+\n+            leg.add_source(url)\n+            leg.add_link(homepage_url)\n+            yield leg\n", "issue": "OH failing since at least 2018-04-18\nOH has been failing since 2018-04-18\n\nBased on automated runs it appears that OH has not run successfully in 2 days (2018-04-18).\n\n\n```\n  02:46:44 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb602/disapprove\n02:46:45 INFO pupa: save bill HB 602 in 132 as bill_cb814bda-43a5-11e8-a12a-020dd716e63e.json\n02:46:45 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/\n02:46:46 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/actions\n02:46:47 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/votes\n02:46:48 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/cmtevotes\n02:46:49 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/veto\n02:46:50 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb603/disapprove\n02:46:51 INFO pupa: save bill HB 603 in 132 as bill_cf14d0f0-43a5-11e8-a12a-020dd716e63e.json\n02:46:51 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/\n02:46:52 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/actions\n02:46:53 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/votes\n02:46:54 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/cmtevotes\n02:46:55 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/veto\n02:46:56 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb604/disapprove\n02:46:57 INFO pupa: save bill HB 604 in 132 as bill_d2b4f550-43a5-11e8-a12a-020dd716e63e.json\n02:46:57 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/\n02:46:58 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/actions\n02:46:59 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/votes\n02:47:00 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/cmtevotes\n02:47:01 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/veto\n02:47:02 INFO scrapelib: GET - http://search-prod.lis.state.oh.us/solarapi/v1/general_assembly_132/bills/hb605/disapprove\n02:47:03 INFO pupa: save bill HB 605 in 132 as bill_d6465452-43a5-11e8-a12a-020dd716e63e.json\n02:47:03 INFO scrapelib: GET - http://www.ohiosenate.gov/members/senate-directory\nloaded Open States pupa settings...\noh (scrape, import)\n  bills: {}\n  people: {}\nTraceback (most recent call last):\n  File \"/opt/openstates/venv-pupa//bin/pupa\", line 11, in <module>\n    load_entry_point('pupa', 'console_scripts', 'pupa')()\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/__main__.py\", line 68, in main\n    subcommands[args.subcommand].handle(args, other)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py\", line 260, in handle\n    return self.do_handle(args, other, juris)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py\", line 305, in do_handle\n    report['scrape'] = self.do_scrape(juris, args, scrapers)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/cli/commands/update.py\", line 173, in do_scrape\n    report[scraper_name] = scraper.do_scrape(**scrape_args)\n  File \"/opt/openstates/venv-pupa/src/pupa/pupa/scrape/base.py\", line 111, in do_scrape\n    for obj in self.scrape(**kwargs) or []:\n  File \"/opt/openstates/openstates/openstates/oh/people.py\", line 48, in scrape\n    yield from self.scrape_page('upper', CHAMBER_URLS['upper'])\n  File \"/opt/openstates/openstates/openstates/oh/people.py\", line 131, in scrape_page\n    page = self.get(url).text\n  File \"/opt/openstates/venv-pupa/lib/python3.5/site-packages/requests/sessions.py\", line 521, in get\n    return self.request('GET', url, **kwargs)\n  File \"/opt/openstates/venv-pupa/lib/python3.5/site-packages/scrapelib/__init__.py\", line 292, in request\n    raise HTTPError(resp)\nscrapelib.HTTPError: 404 while retrieving http://www.ohiosenate.gov/members/senate-directory\n```\n\nVisit http://bobsled.openstates.org for more info.\n\n", "before_files": [{"content": "import re\n\nfrom pupa.scrape import Person, Scraper, Organization\n\nimport lxml.html\n\n\nJOINT_COMMITTEE_OVERRIDE = [  # without Joint\" in the name.\n    \"State Controlling Board\",\n    \"Legislative Service Commission\",\n    \"Correctional Institution Inspection Committee\"\n]\n\nSUBCOMMITTEES = {\n    # The Senate only has Finance subcommittees\n    \"Finance - Corrections Subcommittee\": \"Finance\",\n    \"Finance - Education Subcommittee\": \"Finance\",\n    \"Finance - General Government Subcommittee\": \"Finance\",\n    \"Finance - Higher Ed Subcommittee\": \"Finance\",\n    \"Finance - Workforce Subcommittee\": \"Finance\",\n\n    # The House has mostly Finance, but also one more\n    \"Community and Family Advancement Subcommittee on Minority Affairs\":\n        \"Community and Family Advancement\",\n    \"Finance Subcommittee on Agriculture Development and Natural Resources\":\n        \"Finance\",\n    \"Finance Subcommittee on Health and Human Services\": \"Finance\",\n    \"Finance Subcommittee on Higher Education\": \"Finance\",\n    \"Finance Subcommittee on Primary and Secondary Education\": \"Finance\",\n    \"Finance Subcommittee on Transportation\": \"Finance\",\n    \"Finance Subcommittee on State Government and Agency Review\": \"Finance\"\n}\n\nCHAMBER_URLS = {\n    'upper': \"http://www.ohiosenate.gov/members/senate-directory\",\n    'lower':  \"http://www.ohiohouse.gov/members/member-directory\"\n}\n\ncommittee_cache = {}\n\n\nclass OHLegislatorScraper(Scraper):\n    def scrape(self, chamber=None):\n        self.committees = {}\n        if chamber:\n            yield from self.scrape_page(chamber, CHAMBER_URLS[chamber])\n        else:\n            yield from self.scrape_page('upper', CHAMBER_URLS['upper'])\n            yield from self.scrape_page('lower', CHAMBER_URLS['lower'])\n        yield from self.committees.values()\n\n    def fetch_committee_positions(self, a):\n        page = self.get(a.attrib['href']).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(a.attrib['href'])\n        ret = {}\n        for entry in page.xpath(\"//div[@class='committeeMembers']//td//a\"):\n            person = re.sub(\n                \"\\s+\", \" \", re.sub(\"\\(.*\\)\", \"\", entry.text or \"\")).strip()\n\n            if person == \"\":\n                continue\n\n            title = entry.xpath(\".//div[@class='title']/text()\") or None\n\n            if title:\n                title = title[0]\n                ret[person] = title\n\n        return ret\n\n    def scrape_homepage(self, leg, chamber, homepage):\n        page = self.get(homepage).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(homepage)\n        bio = page.xpath(\n            \"//div[@class='biography']//div[@class='right']//p/text()\")\n        if bio != []:\n            bio = bio[0]\n            leg.extras['biography'] = bio\n\n        fax_line = [\n            x.strip() for x in\n            page.xpath(\n                \"//div[@class='contactModule']/div[@class='data']/text()\"\n            )\n            if \"Fax\" in x\n        ]\n        if fax_line:\n            fax_number = re.search(\n                r'(\\(\\d{3}\\)\\s\\d{3}\\-\\d{4})', fax_line[0]\n            ).group(1)\n            leg.add_contact_detail(type='fax', value=fax_number, note='Capitol Office')\n\n        ctties = page.xpath(\"//div[@class='committeeList']//a\")\n        for a in ctties:\n            entry = a.text_content()\n\n            if entry in committee_cache:\n                committee_positions = committee_cache[entry]\n            else:\n                committee_positions = self.fetch_committee_positions(a)\n                committee_cache[entry] = committee_positions\n\n            chmbr = \"legislature\" if \"joint\" in entry.lower() else chamber\n            if entry in JOINT_COMMITTEE_OVERRIDE:\n                chmbr = \"legislature\"\n\n            kwargs = {}\n\n            if \"subcommittee\" in entry.lower():\n                if entry in SUBCOMMITTEES:\n                    kwargs['subcommittee'] = entry\n                    entry = SUBCOMMITTEES[entry]\n                else:\n                    self.warning(\"No subcommittee known: '%s'\" % (entry))\n                    raise Exception\n            if (chmbr, entry) not in self.committees:\n                org = Organization(\n                    name=entry,\n                    chamber=chmbr,\n                    classification='committee',\n                )\n                self.committees[(chmbr, entry)] = org\n            else:\n                org = self.committees[(chmbr, entry)]\n            org.add_source(homepage)\n            leg.add_membership(org)\n\n    def scrape_page(self, chamber, url):\n        page = self.get(url).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(url)\n\n        for legislator in page.xpath(\n                \"//div[contains(concat(' ', normalize-space(@class), ' '), \"\n                \"' memberModule ')]\"\n                ):\n            img = legislator.xpath(\n                \".//div[@class='thumbnail']//img\")[0].attrib['src']\n            data = legislator.xpath(\".//div[@class='data']\")[0]\n            homepage = data.xpath(\".//a[@class='black']\")[0]\n            full_name = homepage.text_content()\n\n            if \"Vacant\" in full_name:\n                continue\n\n            homepage = homepage.attrib['href']\n            party = data.xpath(\n                \".//span[@class='partyLetter']\")[0].text_content()\n            party = {\"R\": \"Republican\", \"D\": \"Democratic\"}[party]\n            office_lines = data.xpath(\"child::text()\")\n            phone = office_lines.pop(-1)\n            office = \"\\n\".join(office_lines)\n            h3 = data.xpath(\"./h3\")\n            if len(h3):\n                h3 = h3[0]\n                district = h3.xpath(\"./br\")[0].tail.replace(\"District\", \"\"\n                                                            ).strip()\n            else:\n                district = re.findall(\n                    \"\\d+\\.png\",\n                    legislator.attrib['style']\n                )[-1].split(\".\", 1)[0]\n\n            full_name = re.sub(\"\\s+\", \" \", full_name).strip()\n            email = (\n                'rep{0:0{width}}@ohiohouse.gov'\n                if chamber == 'lower' else\n                'sd{0:0{width}}@ohiosenate.gov'\n            ).format(int(district), width=2)\n\n            leg = Person(name=full_name, district=district,\n                         party=party, primary_org=chamber,\n                         image=img)\n\n            leg.add_contact_detail(type='address', value=office, note='Capitol Office')\n            leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')\n            leg.add_contact_detail(type='email', value=email, note='Capitol Office')\n\n            self.scrape_homepage(leg, chamber, homepage)\n\n            leg.add_source(url)\n            leg.add_link(homepage)\n            yield leg\n", "path": "openstates/oh/people.py"}], "after_files": [{"content": "import re\n\nfrom pupa.scrape import Person, Scraper, Organization\n\nimport lxml.html\n\n\nJOINT_COMMITTEE_OVERRIDE = [  # without Joint\" in the name.\n    \"State Controlling Board\",\n    \"Legislative Service Commission\",\n    \"Correctional Institution Inspection Committee\"\n]\n\nSUBCOMMITTEES = {\n    # The Senate only has Finance subcommittees\n    \"Finance - Corrections Subcommittee\": \"Finance\",\n    \"Finance - Education Subcommittee\": \"Finance\",\n    \"Finance - General Government Subcommittee\": \"Finance\",\n    \"Finance - Higher Ed Subcommittee\": \"Finance\",\n    \"Finance - Workforce Subcommittee\": \"Finance\",\n\n    # The House has mostly Finance, but also one more\n    \"Community and Family Advancement Subcommittee on Minority Affairs\":\n        \"Community and Family Advancement\",\n    \"Finance Subcommittee on Agriculture Development and Natural Resources\":\n        \"Finance\",\n    \"Finance Subcommittee on Health and Human Services\": \"Finance\",\n    \"Finance Subcommittee on Higher Education\": \"Finance\",\n    \"Finance Subcommittee on Primary and Secondary Education\": \"Finance\",\n    \"Finance Subcommittee on Transportation\": \"Finance\",\n    \"Finance Subcommittee on State Government and Agency Review\": \"Finance\"\n}\n\nCHAMBER_URLS = {\n    'upper': \"http://www.ohiosenate.gov/senators\",\n    'lower':  \"http://www.ohiohouse.gov/members/member-directory\"\n}\n\ncommittee_cache = {}\n\n\nclass OHLegislatorScraper(Scraper):\n    def scrape(self, chamber=None):\n        self.committees = {}\n        if chamber:\n            yield from self.scrape_page(chamber, CHAMBER_URLS[chamber])\n        else:\n            yield from self.scrape_senator_page('upper', CHAMBER_URLS['upper'])\n            yield from self.scrape_member_page('lower', CHAMBER_URLS['lower'])\n        yield from self.committees.values()\n\n    def fetch_committee_positions(self, a):\n        page = self.get(a.attrib['href']).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(a.attrib['href'])\n        ret = {}\n        for entry in page.xpath(\"//div[@class='committeeMembers']//td//a\"):\n            person = re.sub(\n                \"\\s+\", \" \", re.sub(\"\\(.*\\)\", \"\", entry.text or \"\")).strip()\n\n            if person == \"\":\n                continue\n\n            title = entry.xpath(\".//div[@class='title']/text()\") or None\n\n            if title:\n                title = title[0]\n                ret[person] = title\n\n        return ret\n\n    def scrape_homepage(self, leg, chamber, homepage):\n        page = self.get(homepage).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(homepage)\n        bio = page.xpath(\n            \"//div[@class='biography']//div[@class='right']//p/text()\")\n        if bio != []:\n            bio = bio[0]\n            leg.extras['biography'] = bio\n\n        fax_line = [\n            x.strip() for x in\n            page.xpath(\n                \"//div[@class='contactModule']/div[@class='data']/text()\"\n            )\n            if \"Fax\" in x\n        ]\n        if fax_line:\n            fax_number = re.search(\n                r'(\\(\\d{3}\\)\\s\\d{3}\\-\\d{4})', fax_line[0]\n            ).group(1)\n            leg.add_contact_detail(type='fax', value=fax_number, note='Capitol Office')\n\n        ctties = page.xpath(\"//div[@class='committeeList']//a\")\n        for a in ctties:\n            entry = a.text_content()\n\n            if entry in committee_cache:\n                committee_positions = committee_cache[entry]\n            else:\n                committee_positions = self.fetch_committee_positions(a)\n                committee_cache[entry] = committee_positions\n\n            chmbr = \"legislature\" if \"joint\" in entry.lower() else chamber\n            if entry in JOINT_COMMITTEE_OVERRIDE:\n                chmbr = \"legislature\"\n\n            kwargs = {}\n\n            if \"subcommittee\" in entry.lower():\n                if entry in SUBCOMMITTEES:\n                    kwargs['subcommittee'] = entry\n                    entry = SUBCOMMITTEES[entry]\n                else:\n                    self.warning(\"No subcommittee known: '%s'\" % (entry))\n                    raise Exception\n            if (chmbr, entry) not in self.committees:\n                org = Organization(\n                    name=entry,\n                    chamber=chmbr,\n                    classification='committee',\n                )\n                self.committees[(chmbr, entry)] = org\n            else:\n                org = self.committees[(chmbr, entry)]\n            org.add_source(homepage)\n            leg.add_membership(org)\n\n    def scrape_member_page(self, chamber, url):\n        page = self.get(url).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(url)\n\n        for legislator in page.xpath(\n                \"//div[contains(concat(' ', normalize-space(@class), ' '), \"\n                \"' memberModule ')]\"\n                ):\n            img = legislator.xpath(\n                \".//div[@class='thumbnail']//img\")[0].attrib['src']\n            data = legislator.xpath(\".//div[@class='data']\")[0]\n            homepage = data.xpath(\".//a[@class='black']\")[0]\n            full_name = homepage.text_content()\n\n            if \"Vacant\" in full_name:\n                continue\n\n            homepage = homepage.attrib['href']\n            party = data.xpath(\n                \".//span[@class='partyLetter']\")[0].text_content()\n            party = {\"R\": \"Republican\", \"D\": \"Democratic\"}[party]\n            office_lines = data.xpath(\"child::text()\")\n            phone = office_lines.pop(-1)\n            office = \"\\n\".join(office_lines)\n            h3 = data.xpath(\"./h3\")\n            if len(h3):\n                h3 = h3[0]\n                district = h3.xpath(\"./br\")[0].tail.replace(\"District\", \"\"\n                                                            ).strip()\n            else:\n                district = re.findall(\n                    \"\\d+\\.png\",\n                    legislator.attrib['style']\n                )[-1].split(\".\", 1)[0]\n\n            full_name = re.sub(\"\\s+\", \" \", full_name).strip()\n            email = (\n                'rep{0:0{width}}@ohiohouse.gov'\n                if chamber == 'lower' else\n                'sd{0:0{width}}@ohiosenate.gov'\n            ).format(int(district), width=2)\n\n            leg = Person(name=full_name, district=district,\n                         party=party, primary_org=chamber,\n                         image=img)\n\n            leg.add_contact_detail(type='address', value=office, note='Capitol Office')\n            leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')\n            leg.add_contact_detail(type='email', value=email, note='Capitol Office')\n\n            self.scrape_homepage(leg, chamber, homepage)\n\n            leg.add_source(url)\n            leg.add_link(homepage)\n            yield leg\n\n    def scrape_senator_page(self, chamber, url):\n        page = self.get(url).text\n        page = lxml.html.fromstring(page)\n        page.make_links_absolute(url)\n\n        for legislator in page.xpath(\"//div[contains(concat(' ', normalize-space(@class), ' '), \"\n                                     \"' portraitContainer ')]\"):\n            img = legislator.xpath(\".//div[@class='profileThumbnailBoundingBox']/@style\")[0]\n            img = img[img.find('(')+1:img.find(')')]\n            full_name = legislator.xpath(\".//div[@class='profileName']/a/text()\")[0]\n            homepage_url = legislator.xpath(\".//a[@class='profileImageLink']\")[0].attrib['href']\n            district = legislator.xpath(\".//div[@class='profileDistrict']\"\n                                        \"/a/text()\")[0].split(\"#\")[1]\n\n            if \"Vacant\" in full_name:\n                continue\n\n            homepage = self.get(homepage_url).text\n            page = lxml.html.fromstring(homepage)\n            phone = page.xpath(\"//div[@class='phone']/span/text()\")[0]\n\n            address_lines = page.xpath(\"//div[@class='address']/span/text()\")\n            address = \"\\n\".join(address_lines)\n\n            email = (\n                'rep{0:0{width}}@ohiohouse.gov'\n                if chamber == 'lower' else\n                'sd{0:0{width}}@ohiosenate.gov'\n            ).format(int(district), width=2)\n\n            leg = Person(name=full_name, district=district,\n                         primary_org=chamber, image=img)\n\n            leg.add_contact_detail(type='address', value=address, note='Capitol Office')\n            leg.add_contact_detail(type='voice', value=phone, note='Capitol Office')\n            leg.add_contact_detail(type='email', value=email, note='Capitol Office')\n\n            leg.add_source(url)\n            leg.add_link(homepage_url)\n            yield leg\n", "path": "openstates/oh/people.py"}]}