We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Request serialization should fail for non-picklable objects
The Pickle-based disk queues silently serialize requests that shouldn't be serialized in Python<=3.5. I found this problem when dumping a request with an `ItemLoader` object in its `meta` dict. Python 3.6 fails in [this line](https://github.com/scrapy/scrapy/blob/1.4/scrapy/squeues.py#L27) with `TypeError: can't pickle HtmlElement objects`, because the loader contains a `Selector`, which in turns contains an `HtmlElement` object.

I tested this using the https://github.com/scrapinghub/scrapinghub-stack-scrapy repository, and found that `pickle.loads(pickle.dumps(selector))` doesn't fail, but generates a broken object.

#### Python 2.7, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3)
```
root@04bfc6cf84cd:/# scrapy version -v
Scrapy    : 1.3.3
lxml      : 3.7.2.0
libxml2   : 2.9.3
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 16.6.0
Python    : 2.7.14 (default, Dec 12 2017, 16:55:09) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10
root@04bfc6cf84cd:/# scrapy shell "http://example.org"
2017-12-29 16:49:27 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
(...)
>>> from six.moves import cPickle as pickle
>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))
>>> response.selector.css('a')
[<Selector xpath=u'descendant-or-self::a' data=u'<a href="http://www.iana.org/domains/exa'>]
>>> s2.css('a')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/parsel/selector.py", line 227, in css
    return self.xpath(self._css2xpath(query))
  File "/usr/local/lib/python2.7/site-packages/parsel/selector.py", line 203, in xpath
    **kwargs)
  File "src/lxml/lxml.etree.pyx", line 1584, in lxml.etree._Element.xpath (src/lxml/lxml.etree.c:59349)
  File "src/lxml/xpath.pxi", line 257, in lxml.etree.XPathElementEvaluator.__init__ (src/lxml/lxml.etree.c:170478)
  File "src/lxml/apihelpers.pxi", line 19, in lxml.etree._assertValidNode (src/lxml/lxml.etree.c:16482)
AssertionError: invalid Element proxy at 140144569743064
```


#### Python 3.5, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3-py3)
```
root@1945e2154919:/# scrapy version -v
Scrapy    : 1.3.3
lxml      : 3.7.2.0
libxml2   : 2.9.3
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 16.6.0
Python    : 3.5.4 (default, Dec 12 2017, 16:43:39) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10
root@1945e2154919:/# scrapy shell "http://example.org"
2017-12-29 16:52:37 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
(...)
>>> from six.moves import cPickle as pickle
>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))
>>> response.selector.css('a')
[<Selector xpath='descendant-or-self::a' data='<a href="http://www.iana.org/domains/exa'>]
>>> s2.css('a')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/parsel/selector.py", line 227, in css
    return self.xpath(self._css2xpath(query))
  File "/usr/local/lib/python3.5/site-packages/parsel/selector.py", line 203, in xpath
    **kwargs)
  File "src/lxml/lxml.etree.pyx", line 1584, in lxml.etree._Element.xpath (src/lxml/lxml.etree.c:59349)
  File "src/lxml/xpath.pxi", line 257, in lxml.etree.XPathElementEvaluator.__init__ (src/lxml/lxml.etree.c:170478)
  File "src/lxml/apihelpers.pxi", line 19, in lxml.etree._assertValidNode (src/lxml/lxml.etree.c:16482)
AssertionError: invalid Element proxy at 139862544625976
```


#### Python 3.6, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3-py3)
```
root@43e690443ca7:/# scrapy version -v
Scrapy    : 1.3.3
lxml      : 3.7.2.0
libxml2   : 2.9.3
cssselect : 1.0.1
parsel    : 1.1.0
w3lib     : 1.17.0
Twisted   : 16.6.0
Python    : 3.6.4 (default, Dec 21 2017, 01:35:12) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)
Platform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10
root@43e690443ca7:/# scrapy shell "http://example.org"
2017-12-29 16:54:49 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)
(...)
>>> from six.moves import cPickle as pickle
>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: can't pickle HtmlElement objects
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `scrapy/squeues.py`
Content:
```
1 """
2 Scheduler queues
3 """
4 
5 import marshal
6 from six.moves import cPickle as pickle
7 
8 from queuelib import queue
9 
10 def _serializable_queue(queue_class, serialize, deserialize):
11 
12     class SerializableQueue(queue_class):
13 
14         def push(self, obj):
15             s = serialize(obj)
16             super(SerializableQueue, self).push(s)
17 
18         def pop(self):
19             s = super(SerializableQueue, self).pop()
20             if s:
21                 return deserialize(s)
22 
23     return SerializableQueue
24 
25 def _pickle_serialize(obj):
26     try:
27         return pickle.dumps(obj, protocol=2)
28     # Python>=3.5 raises AttributeError here while
29     # Python<=3.4 raises pickle.PicklingError
30     except (pickle.PicklingError, AttributeError) as e:
31         raise ValueError(str(e))
32 
33 PickleFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \
34     _pickle_serialize, pickle.loads)
35 PickleLifoDiskQueue = _serializable_queue(queue.LifoDiskQueue, \
36     _pickle_serialize, pickle.loads)
37 MarshalFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \
38     marshal.dumps, marshal.loads)
39 MarshalLifoDiskQueue = _serializable_queue(queue.LifoDiskQueue, \
40     marshal.dumps, marshal.loads)
41 FifoMemoryQueue = queue.FifoMemoryQueue
42 LifoMemoryQueue = queue.LifoMemoryQueue
43
```

Path: `setup.py`
Content:
```
1 from os.path import dirname, join
2 from pkg_resources import parse_version
3 from setuptools import setup, find_packages, __version__ as setuptools_version
4 
5 
6 with open(join(dirname(__file__), 'scrapy/VERSION'), 'rb') as f:
7     version = f.read().decode('ascii').strip()
8 
9 
10 def has_environment_marker_platform_impl_support():
11     """Code extracted from 'pytest/setup.py'
12     https://github.com/pytest-dev/pytest/blob/7538680c/setup.py#L31
13 
14     The first known release to support environment marker with range operators
15     it is 18.5, see:
16     https://setuptools.readthedocs.io/en/latest/history.html#id235
17     """
18     return parse_version(setuptools_version) >= parse_version('18.5')
19 
20 
21 extras_require = {}
22 
23 if has_environment_marker_platform_impl_support():
24     extras_require[':platform_python_implementation == "PyPy"'] = [
25         'PyPyDispatcher>=2.1.0',
26     ]
27 
28 
29 setup(
30     name='Scrapy',
31     version=version,
32     url='https://scrapy.org',
33     description='A high-level Web Crawling and Web Scraping framework',
34     long_description=open('README.rst').read(),
35     author='Scrapy developers',
36     maintainer='Pablo Hoffman',
37     maintainer_email='pablo@pablohoffman.com',
38     license='BSD',
39     packages=find_packages(exclude=('tests', 'tests.*')),
40     include_package_data=True,
41     zip_safe=False,
42     entry_points={
43         'console_scripts': ['scrapy = scrapy.cmdline:execute']
44     },
45     classifiers=[
46         'Framework :: Scrapy',
47         'Development Status :: 5 - Production/Stable',
48         'Environment :: Console',
49         'Intended Audience :: Developers',
50         'License :: OSI Approved :: BSD License',
51         'Operating System :: OS Independent',
52         'Programming Language :: Python',
53         'Programming Language :: Python :: 2',
54         'Programming Language :: Python :: 2.7',
55         'Programming Language :: Python :: 3',
56         'Programming Language :: Python :: 3.4',
57         'Programming Language :: Python :: 3.5',
58         'Programming Language :: Python :: 3.6',
59         'Programming Language :: Python :: Implementation :: CPython',
60         'Programming Language :: Python :: Implementation :: PyPy',
61         'Topic :: Internet :: WWW/HTTP',
62         'Topic :: Software Development :: Libraries :: Application Frameworks',
63         'Topic :: Software Development :: Libraries :: Python Modules',
64     ],
65     python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
66     install_requires=[
67         'Twisted>=13.1.0',
68         'w3lib>=1.17.0',
69         'queuelib',
70         'lxml',
71         'pyOpenSSL',
72         'cssselect>=0.9',
73         'six>=1.5.2',
74         'parsel>=1.1',
75         'PyDispatcher>=2.0.5',
76         'service_identity',
77     ],
78     extras_require=extras_require,
79 )
80
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/scrapy/squeues.py b/scrapy/squeues.py
--- a/scrapy/squeues.py
+++ b/scrapy/squeues.py
@@ -25,9 +25,10 @@
 def _pickle_serialize(obj):
     try:
         return pickle.dumps(obj, protocol=2)
-    # Python>=3.5 raises AttributeError here while
-    # Python<=3.4 raises pickle.PicklingError
-    except (pickle.PicklingError, AttributeError) as e:
+    # Python <= 3.4 raises pickle.PicklingError here while
+    # 3.5 <= Python < 3.6 raises AttributeError and
+    # Python >= 3.6 raises TypeError
+    except (pickle.PicklingError, AttributeError, TypeError) as e:
         raise ValueError(str(e))
 
 PickleFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -71,7 +71,7 @@
         'pyOpenSSL',
         'cssselect>=0.9',
         'six>=1.5.2',
-        'parsel>=1.1',
+        'parsel>=1.4',
         'PyDispatcher>=2.0.5',
         'service_identity',
     ],

{"golden_diff": "diff --git a/scrapy/squeues.py b/scrapy/squeues.py\n--- a/scrapy/squeues.py\n+++ b/scrapy/squeues.py\n@@ -25,9 +25,10 @@\n def _pickle_serialize(obj):\n     try:\n         return pickle.dumps(obj, protocol=2)\n-    # Python>=3.5 raises AttributeError here while\n-    # Python<=3.4 raises pickle.PicklingError\n-    except (pickle.PicklingError, AttributeError) as e:\n+    # Python <= 3.4 raises pickle.PicklingError here while\n+    # 3.5 <= Python < 3.6 raises AttributeError and\n+    # Python >= 3.6 raises TypeError\n+    except (pickle.PicklingError, AttributeError, TypeError) as e:\n         raise ValueError(str(e))\n \n PickleFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \\\ndiff --git a/setup.py b/setup.py\n--- a/setup.py\n+++ b/setup.py\n@@ -71,7 +71,7 @@\n         'pyOpenSSL',\n         'cssselect>=0.9',\n         'six>=1.5.2',\n-        'parsel>=1.1',\n+        'parsel>=1.4',\n         'PyDispatcher>=2.0.5',\n         'service_identity',\n     ],\n", "issue": "Request serialization should fail for non-picklable objects\nThe Pickle-based disk queues silently serialize requests that shouldn't be serialized in Python<=3.5. I found this problem when dumping a request with an `ItemLoader` object in its `meta` dict. Python 3.6 fails in [this line](https://github.com/scrapy/scrapy/blob/1.4/scrapy/squeues.py#L27) with `TypeError: can't pickle HtmlElement objects`, because the loader contains a `Selector`, which in turns contains an `HtmlElement` object.\r\n\r\nI tested this using the https://github.com/scrapinghub/scrapinghub-stack-scrapy repository, and found that `pickle.loads(pickle.dumps(selector))` doesn't fail, but generates a broken object.\r\n\r\n#### Python 2.7, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3)\r\n```\r\nroot@04bfc6cf84cd:/# scrapy version -v\r\nScrapy    : 1.3.3\r\nlxml      : 3.7.2.0\r\nlibxml2   : 2.9.3\r\ncssselect : 1.0.1\r\nparsel    : 1.1.0\r\nw3lib     : 1.17.0\r\nTwisted   : 16.6.0\r\nPython    : 2.7.14 (default, Dec 12 2017, 16:55:09) - [GCC 4.9.2]\r\npyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)\r\nPlatform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10\r\nroot@04bfc6cf84cd:/# scrapy shell \"http://example.org\"\r\n2017-12-29 16:49:27 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)\r\n(...)\r\n>>> from six.moves import cPickle as pickle\r\n>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))\r\n>>> response.selector.css('a')\r\n[<Selector xpath=u'descendant-or-self::a' data=u'<a href=\"http://www.iana.org/domains/exa'>]\r\n>>> s2.css('a')\r\nTraceback (most recent call last):\r\n  File \"<console>\", line 1, in <module>\r\n  File \"/usr/local/lib/python2.7/site-packages/parsel/selector.py\", line 227, in css\r\n    return self.xpath(self._css2xpath(query))\r\n  File \"/usr/local/lib/python2.7/site-packages/parsel/selector.py\", line 203, in xpath\r\n    **kwargs)\r\n  File \"src/lxml/lxml.etree.pyx\", line 1584, in lxml.etree._Element.xpath (src/lxml/lxml.etree.c:59349)\r\n  File \"src/lxml/xpath.pxi\", line 257, in lxml.etree.XPathElementEvaluator.__init__ (src/lxml/lxml.etree.c:170478)\r\n  File \"src/lxml/apihelpers.pxi\", line 19, in lxml.etree._assertValidNode (src/lxml/lxml.etree.c:16482)\r\nAssertionError: invalid Element proxy at 140144569743064\r\n```\r\n\r\n\r\n#### Python 3.5, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3-py3)\r\n```\r\nroot@1945e2154919:/# scrapy version -v\r\nScrapy    : 1.3.3\r\nlxml      : 3.7.2.0\r\nlibxml2   : 2.9.3\r\ncssselect : 1.0.1\r\nparsel    : 1.1.0\r\nw3lib     : 1.17.0\r\nTwisted   : 16.6.0\r\nPython    : 3.5.4 (default, Dec 12 2017, 16:43:39) - [GCC 4.9.2]\r\npyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)\r\nPlatform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10\r\nroot@1945e2154919:/# scrapy shell \"http://example.org\"\r\n2017-12-29 16:52:37 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)\r\n(...)\r\n>>> from six.moves import cPickle as pickle\r\n>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))\r\n>>> response.selector.css('a')\r\n[<Selector xpath='descendant-or-self::a' data='<a href=\"http://www.iana.org/domains/exa'>]\r\n>>> s2.css('a')\r\nTraceback (most recent call last):\r\n  File \"<console>\", line 1, in <module>\r\n  File \"/usr/local/lib/python3.5/site-packages/parsel/selector.py\", line 227, in css\r\n    return self.xpath(self._css2xpath(query))\r\n  File \"/usr/local/lib/python3.5/site-packages/parsel/selector.py\", line 203, in xpath\r\n    **kwargs)\r\n  File \"src/lxml/lxml.etree.pyx\", line 1584, in lxml.etree._Element.xpath (src/lxml/lxml.etree.c:59349)\r\n  File \"src/lxml/xpath.pxi\", line 257, in lxml.etree.XPathElementEvaluator.__init__ (src/lxml/lxml.etree.c:170478)\r\n  File \"src/lxml/apihelpers.pxi\", line 19, in lxml.etree._assertValidNode (src/lxml/lxml.etree.c:16482)\r\nAssertionError: invalid Element proxy at 139862544625976\r\n```\r\n\r\n\r\n#### Python 3.6, Scrapy 1.3.3 (https://github.com/scrapinghub/scrapinghub-stack-scrapy/tree/branch-1.3-py3)\r\n```\r\nroot@43e690443ca7:/# scrapy version -v\r\nScrapy    : 1.3.3\r\nlxml      : 3.7.2.0\r\nlibxml2   : 2.9.3\r\ncssselect : 1.0.1\r\nparsel    : 1.1.0\r\nw3lib     : 1.17.0\r\nTwisted   : 16.6.0\r\nPython    : 3.6.4 (default, Dec 21 2017, 01:35:12) - [GCC 4.9.2]\r\npyOpenSSL : 16.2.0 (OpenSSL 1.0.1t  3 May 2016)\r\nPlatform  : Linux-4.9.44-linuxkit-aufs-x86_64-with-debian-8.10\r\nroot@43e690443ca7:/# scrapy shell \"http://example.org\"\r\n2017-12-29 16:54:49 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: scrapybot)\r\n(...)\r\n>>> from six.moves import cPickle as pickle\r\n>>> s2 = pickle.loads(pickle.dumps(response.selector, protocol=2))\r\nTraceback (most recent call last):\r\n  File \"<console>\", line 1, in <module>\r\nTypeError: can't pickle HtmlElement objects\r\n```\n", "before_files": [{"content": "\"\"\"\nScheduler queues\n\"\"\"\n\nimport marshal\nfrom six.moves import cPickle as pickle\n\nfrom queuelib import queue\n\ndef _serializable_queue(queue_class, serialize, deserialize):\n\n    class SerializableQueue(queue_class):\n\n        def push(self, obj):\n            s = serialize(obj)\n            super(SerializableQueue, self).push(s)\n\n        def pop(self):\n            s = super(SerializableQueue, self).pop()\n            if s:\n                return deserialize(s)\n\n    return SerializableQueue\n\ndef _pickle_serialize(obj):\n    try:\n        return pickle.dumps(obj, protocol=2)\n    # Python>=3.5 raises AttributeError here while\n    # Python<=3.4 raises pickle.PicklingError\n    except (pickle.PicklingError, AttributeError) as e:\n        raise ValueError(str(e))\n\nPickleFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \\\n    _pickle_serialize, pickle.loads)\nPickleLifoDiskQueue = _serializable_queue(queue.LifoDiskQueue, \\\n    _pickle_serialize, pickle.loads)\nMarshalFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \\\n    marshal.dumps, marshal.loads)\nMarshalLifoDiskQueue = _serializable_queue(queue.LifoDiskQueue, \\\n    marshal.dumps, marshal.loads)\nFifoMemoryQueue = queue.FifoMemoryQueue\nLifoMemoryQueue = queue.LifoMemoryQueue\n", "path": "scrapy/squeues.py"}, {"content": "from os.path import dirname, join\nfrom pkg_resources import parse_version\nfrom setuptools import setup, find_packages, __version__ as setuptools_version\n\n\nwith open(join(dirname(__file__), 'scrapy/VERSION'), 'rb') as f:\n    version = f.read().decode('ascii').strip()\n\n\ndef has_environment_marker_platform_impl_support():\n    \"\"\"Code extracted from 'pytest/setup.py'\n    https://github.com/pytest-dev/pytest/blob/7538680c/setup.py#L31\n\n    The first known release to support environment marker with range operators\n    it is 18.5, see:\n    https://setuptools.readthedocs.io/en/latest/history.html#id235\n    \"\"\"\n    return parse_version(setuptools_version) >= parse_version('18.5')\n\n\nextras_require = {}\n\nif has_environment_marker_platform_impl_support():\n    extras_require[':platform_python_implementation == \"PyPy\"'] = [\n        'PyPyDispatcher>=2.1.0',\n    ]\n\n\nsetup(\n    name='Scrapy',\n    version=version,\n    url='https://scrapy.org',\n    description='A high-level Web Crawling and Web Scraping framework',\n    long_description=open('README.rst').read(),\n    author='Scrapy developers',\n    maintainer='Pablo Hoffman',\n    maintainer_email='pablo@pablohoffman.com',\n    license='BSD',\n    packages=find_packages(exclude=('tests', 'tests.*')),\n    include_package_data=True,\n    zip_safe=False,\n    entry_points={\n        'console_scripts': ['scrapy = scrapy.cmdline:execute']\n    },\n    classifiers=[\n        'Framework :: Scrapy',\n        'Development Status :: 5 - Production/Stable',\n        'Environment :: Console',\n        'Intended Audience :: Developers',\n        'License :: OSI Approved :: BSD License',\n        'Operating System :: OS Independent',\n        'Programming Language :: Python',\n        'Programming Language :: Python :: 2',\n        'Programming Language :: Python :: 2.7',\n        'Programming Language :: Python :: 3',\n        'Programming Language :: Python :: 3.4',\n        'Programming Language :: Python :: 3.5',\n        'Programming Language :: Python :: 3.6',\n        'Programming Language :: Python :: Implementation :: CPython',\n        'Programming Language :: Python :: Implementation :: PyPy',\n        'Topic :: Internet :: WWW/HTTP',\n        'Topic :: Software Development :: Libraries :: Application Frameworks',\n        'Topic :: Software Development :: Libraries :: Python Modules',\n    ],\n    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',\n    install_requires=[\n        'Twisted>=13.1.0',\n        'w3lib>=1.17.0',\n        'queuelib',\n        'lxml',\n        'pyOpenSSL',\n        'cssselect>=0.9',\n        'six>=1.5.2',\n        'parsel>=1.1',\n        'PyDispatcher>=2.0.5',\n        'service_identity',\n    ],\n    extras_require=extras_require,\n)\n", "path": "setup.py"}], "after_files": [{"content": "\"\"\"\nScheduler queues\n\"\"\"\n\nimport marshal\nfrom six.moves import cPickle as pickle\n\nfrom queuelib import queue\n\ndef _serializable_queue(queue_class, serialize, deserialize):\n\n    class SerializableQueue(queue_class):\n\n        def push(self, obj):\n            s = serialize(obj)\n            super(SerializableQueue, self).push(s)\n\n        def pop(self):\n            s = super(SerializableQueue, self).pop()\n            if s:\n                return deserialize(s)\n\n    return SerializableQueue\n\ndef _pickle_serialize(obj):\n    try:\n        return pickle.dumps(obj, protocol=2)\n    # Python <= 3.4 raises pickle.PicklingError here while\n    # 3.5 <= Python < 3.6 raises AttributeError and\n    # Python >= 3.6 raises TypeError\n    except (pickle.PicklingError, AttributeError, TypeError) as e:\n        raise ValueError(str(e))\n\nPickleFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \\\n    _pickle_serialize, pickle.loads)\nPickleLifoDiskQueue = _serializable_queue(queue.LifoDiskQueue, \\\n    _pickle_serialize, pickle.loads)\nMarshalFifoDiskQueue = _serializable_queue(queue.FifoDiskQueue, \\\n    marshal.dumps, marshal.loads)\nMarshalLifoDiskQueue = _serializable_queue(queue.LifoDiskQueue, \\\n    marshal.dumps, marshal.loads)\nFifoMemoryQueue = queue.FifoMemoryQueue\nLifoMemoryQueue = queue.LifoMemoryQueue\n", "path": "scrapy/squeues.py"}, {"content": "from os.path import dirname, join\nfrom pkg_resources import parse_version\nfrom setuptools import setup, find_packages, __version__ as setuptools_version\n\n\nwith open(join(dirname(__file__), 'scrapy/VERSION'), 'rb') as f:\n    version = f.read().decode('ascii').strip()\n\n\ndef has_environment_marker_platform_impl_support():\n    \"\"\"Code extracted from 'pytest/setup.py'\n    https://github.com/pytest-dev/pytest/blob/7538680c/setup.py#L31\n\n    The first known release to support environment marker with range operators\n    it is 18.5, see:\n    https://setuptools.readthedocs.io/en/latest/history.html#id235\n    \"\"\"\n    return parse_version(setuptools_version) >= parse_version('18.5')\n\n\nextras_require = {}\n\nif has_environment_marker_platform_impl_support():\n    extras_require[':platform_python_implementation == \"PyPy\"'] = [\n        'PyPyDispatcher>=2.1.0',\n    ]\n\n\nsetup(\n    name='Scrapy',\n    version=version,\n    url='https://scrapy.org',\n    description='A high-level Web Crawling and Web Scraping framework',\n    long_description=open('README.rst').read(),\n    author='Scrapy developers',\n    maintainer='Pablo Hoffman',\n    maintainer_email='pablo@pablohoffman.com',\n    license='BSD',\n    packages=find_packages(exclude=('tests', 'tests.*')),\n    include_package_data=True,\n    zip_safe=False,\n    entry_points={\n        'console_scripts': ['scrapy = scrapy.cmdline:execute']\n    },\n    classifiers=[\n        'Framework :: Scrapy',\n        'Development Status :: 5 - Production/Stable',\n        'Environment :: Console',\n        'Intended Audience :: Developers',\n        'License :: OSI Approved :: BSD License',\n        'Operating System :: OS Independent',\n        'Programming Language :: Python',\n        'Programming Language :: Python :: 2',\n        'Programming Language :: Python :: 2.7',\n        'Programming Language :: Python :: 3',\n        'Programming Language :: Python :: 3.4',\n        'Programming Language :: Python :: 3.5',\n        'Programming Language :: Python :: 3.6',\n        'Programming Language :: Python :: Implementation :: CPython',\n        'Programming Language :: Python :: Implementation :: PyPy',\n        'Topic :: Internet :: WWW/HTTP',\n        'Topic :: Software Development :: Libraries :: Application Frameworks',\n        'Topic :: Software Development :: Libraries :: Python Modules',\n    ],\n    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',\n    install_requires=[\n        'Twisted>=13.1.0',\n        'w3lib>=1.17.0',\n        'queuelib',\n        'lxml',\n        'pyOpenSSL',\n        'cssselect>=0.9',\n        'six>=1.5.2',\n        'parsel>=1.4',\n        'PyDispatcher>=2.0.5',\n        'service_identity',\n    ],\n    extras_require=extras_require,\n)\n", "path": "setup.py"}]}