We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
CachingHostnameResolver does not work with reactor.resolve()
### Description

Hi. Thank you for maintaining this awesome software :)

I am working on a project using scrapy that implements a custom downloader class ([link](https://github.com/michael-lazar/mozz-archiver/blob/master/mozz_archiver/downloaders.py)).

I want to resolve IPv6 addresses, and I found the section in the documentation about the ``DNS_RESOLVER`` setting that was added in #4227. I tried enabling the new ``DNS_RESOLVER = "scrapy.resolver.CachingHostnameResolver"`` and was immediately greeted with this exception.

```
Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/scrapy/commands/crawl.py", line 27, in run
    self.crawler_process.start()
  File "/usr/local/lib/python3.8/site-packages/scrapy/crawler.py", line 327, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/base.py", line 1283, in run
    self.mainLoop()
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/base.py", line 1292, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/base.py", line 913, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/tcp.py", line 449, in resolveAddress
    d = self.reactor.resolve(self.addr[0])
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/base.py", line 638, in resolve
    return self.resolver.getHostByName(name, timeout)
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/_resolver.py", line 277, in getHostByName
    self._nameResolver.resolveHostName(FirstOneWins(result), name, 0,
  File "/usr/local/lib/python3.8/site-packages/scrapy/resolver.py", line 80, in resolveHostName
    class CachingResolutionReceiver(resolutionReceiver):
builtins.TypeError: __init__() takes 2 positional arguments but 4 were given
```

### Steps to Reproduce

This is also reproducible using the bundled FTP downloader

1. ``scrapy startproject scrapy_test``
2. ``scrapy genspider example mozz.us``
3. Add ``DNS_RESOLVER = "scrapy.resolver.CachingHostnameResolver"`` to the settings file
4. Change the spider start_url to ``ftp://mozz.us``
5. ``scrapy crawl scrapy_test``

### Versions

```
Scrapy       : 2.3.0
lxml         : 4.5.2.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 20.3.0
Python       : 3.8.5 (default, Jul 21 2020, 10:48:26) - [Clang 11.0.3 (clang-1103.0.32.62)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020)
cryptography : 3.0
Platform     : macOS-10.15.6-x86_64-i386-64bit
```

### Additional context

This was a tricky one to debug because everything works as expected with the HTTP Agent downloader. This issue only appears when you implement a downloader that depends on calling ``reactor.resolve()`` directly without using ``twisted.internet.endpoints.HostnameEndpoint``.

I discovered that in the twisted [IHostnameResolver](https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IHostnameResolver.html) interface, the ``resolutionReceiver`` method argument is expected to be an *instance* of a resolution receiver class, and not a *type* of a resolution receiver class. So I believe the scrapy code below is incorrect:

https://github.com/scrapy/scrapy/blob/5e997587d9b13344a0afa9bb4cf781829a66ce23/scrapy/resolver.py#L76-L80

The subclass here only works with the Scrapy Agent because the ``HostnameEndpoint`` does this weird thing where it defines a class with only static methods, so it can pass the class itself instead of instantiating it.

https://github.com/twisted/twisted/blob/22f949f7ce187513f0c218b73186c8a73baa00b4/src/twisted/internet/endpoints.py#L942-L958

```python
        @provider(IResolutionReceiver)
        class EndpointReceiver:
            @staticmethod
            def resolutionBegan(resolutionInProgress):
                pass

            @staticmethod
            def addressResolved(address):
                addresses.append(address)

            @staticmethod
            def resolutionComplete():
                d.callback(addresses)

        self._nameResolver.resolveHostName(
            EndpointReceiver, self._hostText, portNumber=self._port
        )
```

However, there are other places in the twisted reactor where twisted does pass an object instance directly to this method.

https://github.com/twisted/twisted/blob/7e3ce790ca9f004ab386f9ecbba8f505d66cd3bd/src/twisted/internet/_resolver.py#L307

```python
        result = Deferred()
        self._nameResolver.resolveHostName(FirstOneWins(result), name, 0, [IPv4Address])
        return result
```




--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `scrapy/resolver.py`
Content:
```
1 from twisted.internet import defer
2 from twisted.internet.base import ThreadedResolver
3 from twisted.internet.interfaces import IHostnameResolver, IResolutionReceiver, IResolverSimple
4 from zope.interface.declarations import implementer, provider
5 
6 from scrapy.utils.datatypes import LocalCache
7 
8 
9 # TODO: cache misses
10 dnscache = LocalCache(10000)
11 
12 
13 @implementer(IResolverSimple)
14 class CachingThreadedResolver(ThreadedResolver):
15     """
16     Default caching resolver. IPv4 only, supports setting a timeout value for DNS requests.
17     """
18 
19     def __init__(self, reactor, cache_size, timeout):
20         super().__init__(reactor)
21         dnscache.limit = cache_size
22         self.timeout = timeout
23 
24     @classmethod
25     def from_crawler(cls, crawler, reactor):
26         if crawler.settings.getbool('DNSCACHE_ENABLED'):
27             cache_size = crawler.settings.getint('DNSCACHE_SIZE')
28         else:
29             cache_size = 0
30         return cls(reactor, cache_size, crawler.settings.getfloat('DNS_TIMEOUT'))
31 
32     def install_on_reactor(self):
33         self.reactor.installResolver(self)
34 
35     def getHostByName(self, name, timeout=None):
36         if name in dnscache:
37             return defer.succeed(dnscache[name])
38         # in Twisted<=16.6, getHostByName() is always called with
39         # a default timeout of 60s (actually passed as (1, 3, 11, 45) tuple),
40         # so the input argument above is simply overridden
41         # to enforce Scrapy's DNS_TIMEOUT setting's value
42         timeout = (self.timeout,)
43         d = super().getHostByName(name, timeout)
44         if dnscache.limit:
45             d.addCallback(self._cache_result, name)
46         return d
47 
48     def _cache_result(self, result, name):
49         dnscache[name] = result
50         return result
51 
52 
53 @implementer(IHostnameResolver)
54 class CachingHostnameResolver:
55     """
56     Experimental caching resolver. Resolves IPv4 and IPv6 addresses,
57     does not support setting a timeout value for DNS requests.
58     """
59 
60     def __init__(self, reactor, cache_size):
61         self.reactor = reactor
62         self.original_resolver = reactor.nameResolver
63         dnscache.limit = cache_size
64 
65     @classmethod
66     def from_crawler(cls, crawler, reactor):
67         if crawler.settings.getbool('DNSCACHE_ENABLED'):
68             cache_size = crawler.settings.getint('DNSCACHE_SIZE')
69         else:
70             cache_size = 0
71         return cls(reactor, cache_size)
72 
73     def install_on_reactor(self):
74         self.reactor.installNameResolver(self)
75 
76     def resolveHostName(self, resolutionReceiver, hostName, portNumber=0,
77                         addressTypes=None, transportSemantics='TCP'):
78 
79         @provider(IResolutionReceiver)
80         class CachingResolutionReceiver(resolutionReceiver):
81 
82             def resolutionBegan(self, resolution):
83                 super().resolutionBegan(resolution)
84                 self.resolution = resolution
85                 self.resolved = False
86 
87             def addressResolved(self, address):
88                 super().addressResolved(address)
89                 self.resolved = True
90 
91             def resolutionComplete(self):
92                 super().resolutionComplete()
93                 if self.resolved:
94                     dnscache[hostName] = self.resolution
95 
96         try:
97             return dnscache[hostName]
98         except KeyError:
99             return self.original_resolver.resolveHostName(
100                 CachingResolutionReceiver(),
101                 hostName,
102                 portNumber,
103                 addressTypes,
104                 transportSemantics
105             )
106
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/scrapy/resolver.py b/scrapy/resolver.py
--- a/scrapy/resolver.py
+++ b/scrapy/resolver.py
@@ -1,6 +1,6 @@
 from twisted.internet import defer
 from twisted.internet.base import ThreadedResolver
-from twisted.internet.interfaces import IHostnameResolver, IResolutionReceiver, IResolverSimple
+from twisted.internet.interfaces import IHostResolution, IHostnameResolver, IResolutionReceiver, IResolverSimple
 from zope.interface.declarations import implementer, provider
 
 from scrapy.utils.datatypes import LocalCache
@@ -50,6 +50,36 @@
         return result
 
 
+@implementer(IHostResolution)
+class HostResolution:
+    def __init__(self, name):
+        self.name = name
+
+    def cancel(self):
+        raise NotImplementedError()
+
+
+@provider(IResolutionReceiver)
+class _CachingResolutionReceiver:
+    def __init__(self, resolutionReceiver, hostName):
+        self.resolutionReceiver = resolutionReceiver
+        self.hostName = hostName
+        self.addresses = []
+
+    def resolutionBegan(self, resolution):
+        self.resolutionReceiver.resolutionBegan(resolution)
+        self.resolution = resolution
+
+    def addressResolved(self, address):
+        self.resolutionReceiver.addressResolved(address)
+        self.addresses.append(address)
+
+    def resolutionComplete(self):
+        self.resolutionReceiver.resolutionComplete()
+        if self.addresses:
+            dnscache[self.hostName] = self.addresses
+
+
 @implementer(IHostnameResolver)
 class CachingHostnameResolver:
     """
@@ -73,33 +103,22 @@
     def install_on_reactor(self):
         self.reactor.installNameResolver(self)
 
-    def resolveHostName(self, resolutionReceiver, hostName, portNumber=0,
-                        addressTypes=None, transportSemantics='TCP'):
-
-        @provider(IResolutionReceiver)
-        class CachingResolutionReceiver(resolutionReceiver):
-
-            def resolutionBegan(self, resolution):
-                super().resolutionBegan(resolution)
-                self.resolution = resolution
-                self.resolved = False
-
-            def addressResolved(self, address):
-                super().addressResolved(address)
-                self.resolved = True
-
-            def resolutionComplete(self):
-                super().resolutionComplete()
-                if self.resolved:
-                    dnscache[hostName] = self.resolution
-
+    def resolveHostName(
+        self, resolutionReceiver, hostName, portNumber=0, addressTypes=None, transportSemantics="TCP"
+    ):
         try:
-            return dnscache[hostName]
+            addresses = dnscache[hostName]
         except KeyError:
             return self.original_resolver.resolveHostName(
-                CachingResolutionReceiver(),
+                _CachingResolutionReceiver(resolutionReceiver, hostName),
                 hostName,
                 portNumber,
                 addressTypes,
-                transportSemantics
+                transportSemantics,
             )
+        else:
+            resolutionReceiver.resolutionBegan(HostResolution(hostName))
+            for addr in addresses:
+                resolutionReceiver.addressResolved(addr)
+            resolutionReceiver.resolutionComplete()
+            return resolutionReceiver

{"golden_diff": "diff --git a/scrapy/resolver.py b/scrapy/resolver.py\n--- a/scrapy/resolver.py\n+++ b/scrapy/resolver.py\n@@ -1,6 +1,6 @@\n from twisted.internet import defer\n from twisted.internet.base import ThreadedResolver\n-from twisted.internet.interfaces import IHostnameResolver, IResolutionReceiver, IResolverSimple\n+from twisted.internet.interfaces import IHostResolution, IHostnameResolver, IResolutionReceiver, IResolverSimple\n from zope.interface.declarations import implementer, provider\n \n from scrapy.utils.datatypes import LocalCache\n@@ -50,6 +50,36 @@\n         return result\n \n \n+@implementer(IHostResolution)\n+class HostResolution:\n+    def __init__(self, name):\n+        self.name = name\n+\n+    def cancel(self):\n+        raise NotImplementedError()\n+\n+\n+@provider(IResolutionReceiver)\n+class _CachingResolutionReceiver:\n+    def __init__(self, resolutionReceiver, hostName):\n+        self.resolutionReceiver = resolutionReceiver\n+        self.hostName = hostName\n+        self.addresses = []\n+\n+    def resolutionBegan(self, resolution):\n+        self.resolutionReceiver.resolutionBegan(resolution)\n+        self.resolution = resolution\n+\n+    def addressResolved(self, address):\n+        self.resolutionReceiver.addressResolved(address)\n+        self.addresses.append(address)\n+\n+    def resolutionComplete(self):\n+        self.resolutionReceiver.resolutionComplete()\n+        if self.addresses:\n+            dnscache[self.hostName] = self.addresses\n+\n+\n @implementer(IHostnameResolver)\n class CachingHostnameResolver:\n     \"\"\"\n@@ -73,33 +103,22 @@\n     def install_on_reactor(self):\n         self.reactor.installNameResolver(self)\n \n-    def resolveHostName(self, resolutionReceiver, hostName, portNumber=0,\n-                        addressTypes=None, transportSemantics='TCP'):\n-\n-        @provider(IResolutionReceiver)\n-        class CachingResolutionReceiver(resolutionReceiver):\n-\n-            def resolutionBegan(self, resolution):\n-                super().resolutionBegan(resolution)\n-                self.resolution = resolution\n-                self.resolved = False\n-\n-            def addressResolved(self, address):\n-                super().addressResolved(address)\n-                self.resolved = True\n-\n-            def resolutionComplete(self):\n-                super().resolutionComplete()\n-                if self.resolved:\n-                    dnscache[hostName] = self.resolution\n-\n+    def resolveHostName(\n+        self, resolutionReceiver, hostName, portNumber=0, addressTypes=None, transportSemantics=\"TCP\"\n+    ):\n         try:\n-            return dnscache[hostName]\n+            addresses = dnscache[hostName]\n         except KeyError:\n             return self.original_resolver.resolveHostName(\n-                CachingResolutionReceiver(),\n+                _CachingResolutionReceiver(resolutionReceiver, hostName),\n                 hostName,\n                 portNumber,\n                 addressTypes,\n-                transportSemantics\n+                transportSemantics,\n             )\n+        else:\n+            resolutionReceiver.resolutionBegan(HostResolution(hostName))\n+            for addr in addresses:\n+                resolutionReceiver.addressResolved(addr)\n+            resolutionReceiver.resolutionComplete()\n+            return resolutionReceiver\n", "issue": "CachingHostnameResolver does not work with reactor.resolve()\n### Description\r\n\r\nHi. Thank you for maintaining this awesome software :)\r\n\r\nI am working on a project using scrapy that implements a custom downloader class ([link](https://github.com/michael-lazar/mozz-archiver/blob/master/mozz_archiver/downloaders.py)).\r\n\r\nI want to resolve IPv6 addresses, and I found the section in the documentation about the ``DNS_RESOLVER`` setting that was added in #4227. I tried enabling the new ``DNS_RESOLVER = \"scrapy.resolver.CachingHostnameResolver\"`` and was immediately greeted with this exception.\r\n\r\n```\r\nUnhandled Error\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python3.8/site-packages/scrapy/commands/crawl.py\", line 27, in run\r\n    self.crawler_process.start()\r\n  File \"/usr/local/lib/python3.8/site-packages/scrapy/crawler.py\", line 327, in start\r\n    reactor.run(installSignalHandlers=False)  # blocking call\r\n  File \"/usr/local/lib/python3.8/site-packages/twisted/internet/base.py\", line 1283, in run\r\n    self.mainLoop()\r\n  File \"/usr/local/lib/python3.8/site-packages/twisted/internet/base.py\", line 1292, in mainLoop\r\n    self.runUntilCurrent()\r\n--- <exception caught here> ---\r\n  File \"/usr/local/lib/python3.8/site-packages/twisted/internet/base.py\", line 913, in runUntilCurrent\r\n    call.func(*call.args, **call.kw)\r\n  File \"/usr/local/lib/python3.8/site-packages/twisted/internet/tcp.py\", line 449, in resolveAddress\r\n    d = self.reactor.resolve(self.addr[0])\r\n  File \"/usr/local/lib/python3.8/site-packages/twisted/internet/base.py\", line 638, in resolve\r\n    return self.resolver.getHostByName(name, timeout)\r\n  File \"/usr/local/lib/python3.8/site-packages/twisted/internet/_resolver.py\", line 277, in getHostByName\r\n    self._nameResolver.resolveHostName(FirstOneWins(result), name, 0,\r\n  File \"/usr/local/lib/python3.8/site-packages/scrapy/resolver.py\", line 80, in resolveHostName\r\n    class CachingResolutionReceiver(resolutionReceiver):\r\nbuiltins.TypeError: __init__() takes 2 positional arguments but 4 were given\r\n```\r\n\r\n### Steps to Reproduce\r\n\r\nThis is also reproducible using the bundled FTP downloader\r\n\r\n1. ``scrapy startproject scrapy_test``\r\n2. ``scrapy genspider example mozz.us``\r\n3. Add ``DNS_RESOLVER = \"scrapy.resolver.CachingHostnameResolver\"`` to the settings file\r\n4. Change the spider start_url to ``ftp://mozz.us``\r\n5. ``scrapy crawl scrapy_test``\r\n\r\n### Versions\r\n\r\n```\r\nScrapy       : 2.3.0\r\nlxml         : 4.5.2.0\r\nlibxml2      : 2.9.10\r\ncssselect    : 1.1.0\r\nparsel       : 1.6.0\r\nw3lib        : 1.22.0\r\nTwisted      : 20.3.0\r\nPython       : 3.8.5 (default, Jul 21 2020, 10:48:26) - [Clang 11.0.3 (clang-1103.0.32.62)]\r\npyOpenSSL    : 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020)\r\ncryptography : 3.0\r\nPlatform     : macOS-10.15.6-x86_64-i386-64bit\r\n```\r\n\r\n### Additional context\r\n\r\nThis was a tricky one to debug because everything works as expected with the HTTP Agent downloader. This issue only appears when you implement a downloader that depends on calling ``reactor.resolve()`` directly without using ``twisted.internet.endpoints.HostnameEndpoint``.\r\n\r\nI discovered that in the twisted [IHostnameResolver](https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IHostnameResolver.html) interface, the ``resolutionReceiver`` method argument is expected to be an *instance* of a resolution receiver class, and not a *type* of a resolution receiver class. So I believe the scrapy code below is incorrect:\r\n\r\nhttps://github.com/scrapy/scrapy/blob/5e997587d9b13344a0afa9bb4cf781829a66ce23/scrapy/resolver.py#L76-L80\r\n\r\nThe subclass here only works with the Scrapy Agent because the ``HostnameEndpoint`` does this weird thing where it defines a class with only static methods, so it can pass the class itself instead of instantiating it.\r\n\r\nhttps://github.com/twisted/twisted/blob/22f949f7ce187513f0c218b73186c8a73baa00b4/src/twisted/internet/endpoints.py#L942-L958\r\n\r\n```python\r\n        @provider(IResolutionReceiver)\r\n        class EndpointReceiver:\r\n            @staticmethod\r\n            def resolutionBegan(resolutionInProgress):\r\n                pass\r\n\r\n            @staticmethod\r\n            def addressResolved(address):\r\n                addresses.append(address)\r\n\r\n            @staticmethod\r\n            def resolutionComplete():\r\n                d.callback(addresses)\r\n\r\n        self._nameResolver.resolveHostName(\r\n            EndpointReceiver, self._hostText, portNumber=self._port\r\n        )\r\n```\r\n\r\nHowever, there are other places in the twisted reactor where twisted does pass an object instance directly to this method.\r\n\r\nhttps://github.com/twisted/twisted/blob/7e3ce790ca9f004ab386f9ecbba8f505d66cd3bd/src/twisted/internet/_resolver.py#L307\r\n\r\n```python\r\n        result = Deferred()\r\n        self._nameResolver.resolveHostName(FirstOneWins(result), name, 0, [IPv4Address])\r\n        return result\r\n```\r\n\r\n\r\n\n", "before_files": [{"content": "from twisted.internet import defer\nfrom twisted.internet.base import ThreadedResolver\nfrom twisted.internet.interfaces import IHostnameResolver, IResolutionReceiver, IResolverSimple\nfrom zope.interface.declarations import implementer, provider\n\nfrom scrapy.utils.datatypes import LocalCache\n\n\n# TODO: cache misses\ndnscache = LocalCache(10000)\n\n\n@implementer(IResolverSimple)\nclass CachingThreadedResolver(ThreadedResolver):\n    \"\"\"\n    Default caching resolver. IPv4 only, supports setting a timeout value for DNS requests.\n    \"\"\"\n\n    def __init__(self, reactor, cache_size, timeout):\n        super().__init__(reactor)\n        dnscache.limit = cache_size\n        self.timeout = timeout\n\n    @classmethod\n    def from_crawler(cls, crawler, reactor):\n        if crawler.settings.getbool('DNSCACHE_ENABLED'):\n            cache_size = crawler.settings.getint('DNSCACHE_SIZE')\n        else:\n            cache_size = 0\n        return cls(reactor, cache_size, crawler.settings.getfloat('DNS_TIMEOUT'))\n\n    def install_on_reactor(self):\n        self.reactor.installResolver(self)\n\n    def getHostByName(self, name, timeout=None):\n        if name in dnscache:\n            return defer.succeed(dnscache[name])\n        # in Twisted<=16.6, getHostByName() is always called with\n        # a default timeout of 60s (actually passed as (1, 3, 11, 45) tuple),\n        # so the input argument above is simply overridden\n        # to enforce Scrapy's DNS_TIMEOUT setting's value\n        timeout = (self.timeout,)\n        d = super().getHostByName(name, timeout)\n        if dnscache.limit:\n            d.addCallback(self._cache_result, name)\n        return d\n\n    def _cache_result(self, result, name):\n        dnscache[name] = result\n        return result\n\n\n@implementer(IHostnameResolver)\nclass CachingHostnameResolver:\n    \"\"\"\n    Experimental caching resolver. Resolves IPv4 and IPv6 addresses,\n    does not support setting a timeout value for DNS requests.\n    \"\"\"\n\n    def __init__(self, reactor, cache_size):\n        self.reactor = reactor\n        self.original_resolver = reactor.nameResolver\n        dnscache.limit = cache_size\n\n    @classmethod\n    def from_crawler(cls, crawler, reactor):\n        if crawler.settings.getbool('DNSCACHE_ENABLED'):\n            cache_size = crawler.settings.getint('DNSCACHE_SIZE')\n        else:\n            cache_size = 0\n        return cls(reactor, cache_size)\n\n    def install_on_reactor(self):\n        self.reactor.installNameResolver(self)\n\n    def resolveHostName(self, resolutionReceiver, hostName, portNumber=0,\n                        addressTypes=None, transportSemantics='TCP'):\n\n        @provider(IResolutionReceiver)\n        class CachingResolutionReceiver(resolutionReceiver):\n\n            def resolutionBegan(self, resolution):\n                super().resolutionBegan(resolution)\n                self.resolution = resolution\n                self.resolved = False\n\n            def addressResolved(self, address):\n                super().addressResolved(address)\n                self.resolved = True\n\n            def resolutionComplete(self):\n                super().resolutionComplete()\n                if self.resolved:\n                    dnscache[hostName] = self.resolution\n\n        try:\n            return dnscache[hostName]\n        except KeyError:\n            return self.original_resolver.resolveHostName(\n                CachingResolutionReceiver(),\n                hostName,\n                portNumber,\n                addressTypes,\n                transportSemantics\n            )\n", "path": "scrapy/resolver.py"}], "after_files": [{"content": "from twisted.internet import defer\nfrom twisted.internet.base import ThreadedResolver\nfrom twisted.internet.interfaces import IHostResolution, IHostnameResolver, IResolutionReceiver, IResolverSimple\nfrom zope.interface.declarations import implementer, provider\n\nfrom scrapy.utils.datatypes import LocalCache\n\n\n# TODO: cache misses\ndnscache = LocalCache(10000)\n\n\n@implementer(IResolverSimple)\nclass CachingThreadedResolver(ThreadedResolver):\n    \"\"\"\n    Default caching resolver. IPv4 only, supports setting a timeout value for DNS requests.\n    \"\"\"\n\n    def __init__(self, reactor, cache_size, timeout):\n        super().__init__(reactor)\n        dnscache.limit = cache_size\n        self.timeout = timeout\n\n    @classmethod\n    def from_crawler(cls, crawler, reactor):\n        if crawler.settings.getbool('DNSCACHE_ENABLED'):\n            cache_size = crawler.settings.getint('DNSCACHE_SIZE')\n        else:\n            cache_size = 0\n        return cls(reactor, cache_size, crawler.settings.getfloat('DNS_TIMEOUT'))\n\n    def install_on_reactor(self):\n        self.reactor.installResolver(self)\n\n    def getHostByName(self, name, timeout=None):\n        if name in dnscache:\n            return defer.succeed(dnscache[name])\n        # in Twisted<=16.6, getHostByName() is always called with\n        # a default timeout of 60s (actually passed as (1, 3, 11, 45) tuple),\n        # so the input argument above is simply overridden\n        # to enforce Scrapy's DNS_TIMEOUT setting's value\n        timeout = (self.timeout,)\n        d = super().getHostByName(name, timeout)\n        if dnscache.limit:\n            d.addCallback(self._cache_result, name)\n        return d\n\n    def _cache_result(self, result, name):\n        dnscache[name] = result\n        return result\n\n\n@implementer(IHostResolution)\nclass HostResolution:\n    def __init__(self, name):\n        self.name = name\n\n    def cancel(self):\n        raise NotImplementedError()\n\n\n@provider(IResolutionReceiver)\nclass _CachingResolutionReceiver:\n    def __init__(self, resolutionReceiver, hostName):\n        self.resolutionReceiver = resolutionReceiver\n        self.hostName = hostName\n        self.addresses = []\n\n    def resolutionBegan(self, resolution):\n        self.resolutionReceiver.resolutionBegan(resolution)\n        self.resolution = resolution\n\n    def addressResolved(self, address):\n        self.resolutionReceiver.addressResolved(address)\n        self.addresses.append(address)\n\n    def resolutionComplete(self):\n        self.resolutionReceiver.resolutionComplete()\n        if self.addresses:\n            dnscache[self.hostName] = self.addresses\n\n\n@implementer(IHostnameResolver)\nclass CachingHostnameResolver:\n    \"\"\"\n    Experimental caching resolver. Resolves IPv4 and IPv6 addresses,\n    does not support setting a timeout value for DNS requests.\n    \"\"\"\n\n    def __init__(self, reactor, cache_size):\n        self.reactor = reactor\n        self.original_resolver = reactor.nameResolver\n        dnscache.limit = cache_size\n\n    @classmethod\n    def from_crawler(cls, crawler, reactor):\n        if crawler.settings.getbool('DNSCACHE_ENABLED'):\n            cache_size = crawler.settings.getint('DNSCACHE_SIZE')\n        else:\n            cache_size = 0\n        return cls(reactor, cache_size)\n\n    def install_on_reactor(self):\n        self.reactor.installNameResolver(self)\n\n    def resolveHostName(\n        self, resolutionReceiver, hostName, portNumber=0, addressTypes=None, transportSemantics=\"TCP\"\n    ):\n        try:\n            addresses = dnscache[hostName]\n        except KeyError:\n            return self.original_resolver.resolveHostName(\n                _CachingResolutionReceiver(resolutionReceiver, hostName),\n                hostName,\n                portNumber,\n                addressTypes,\n                transportSemantics,\n            )\n        else:\n            resolutionReceiver.resolutionBegan(HostResolution(hostName))\n            for addr in addresses:\n                resolutionReceiver.addressResolved(addr)\n            resolutionReceiver.resolutionComplete()\n            return resolutionReceiver\n", "path": "scrapy/resolver.py"}]}