We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[BUG] DistutilsMetaFinder breaks distutils imports when old setuptools is higher on path
### setuptools version

setuptools==60.0.4

### Python version

python3.8

### OS

RHEL8, FreeBSD12/13, Ubuntu 20.04

### Additional environment information

Any environment where an old setuptools (eg OS package) is higher in `sys.path`, but a newer setuptools is installed anywhere on `sys.path` (with its distutils shimming `.pth` file active)

### Description

Any import of `distutils` will bomb when a new setuptools is present, but an older setuptools is higher on the path (since the new setuptools' path shim blindly tries to import from `setuptools._distutils`, which doesn't exist in old versions of setuptools). This has probably been the case ever since the distutils shim was created, but now that it's active by default, it's a bigger problem.

The place we hit this was running `pip install` from a venv with `--system-site-packages`- pip's build isolation installs latest `setuptools`, but then shuffles `sys.path` around such that the ancient OS-packaged `setuptools` is higher on the path. The first thing that tries to import `distutils` invokes the metapath finder, which has no validation or recovery and just lets the `ModuleNotFoundError` fly.

At first blush, it seems like the metapath shim's `spec_for_distutils` should try to at least verify the co-location of `setuptools` with the shim impl (ie, that they're under the same path prefix) and/or that the `_distutils` subpackage is available via `find_spec` before it actually imports it. Mixing and matching the shim version with random setuptools versions seems like a recipe for trouble down the road...

I could probably throw together a PR for this if there's a consensus that this approach is sane.

### Expected behavior

The presence of a new-but-not-loadable setuptools on sys.path doesn't break old setuptools/distutils.


### How to Reproduce

(NB: this is a much simpler repro than the OS installed setuptools that's actually failing, but illustrates the problem more easily on any OS)
```
python3 -m venv /tmp/bang && source /tmp/bang/bin/activate  # give us a clean venv to work in
python -m pip install 'setuptools>60'  # install new setuptools to venv site-packages
python -m pip install 'setuptools<45' -t .  # install old setuptools to cwd- higher than site-packages on path for most systems
python -c 'import distutils'
```

### Output

```console
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/bang/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/mdavis/setuptools/__init__.py", line 6, in <module>
    import distutils.core
  File "/tmp/bang/lib64/python3.9/site-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools._distutils'

```


### Code of Conduct

- [X] I agree to follow the PSF Code of Conduct

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `_distutils_hack/__init__.py`
Content:
```
1 import sys
2 import os
3 import re
4 import importlib
5 import warnings
6 import contextlib
7 
8 
9 is_pypy = '__pypy__' in sys.builtin_module_names
10 
11 
12 warnings.filterwarnings('ignore',
13                         r'.+ distutils\b.+ deprecated',
14                         DeprecationWarning)
15 
16 
17 def warn_distutils_present():
18     if 'distutils' not in sys.modules:
19         return
20     if is_pypy and sys.version_info < (3, 7):
21         # PyPy for 3.6 unconditionally imports distutils, so bypass the warning
22         # https://foss.heptapod.net/pypy/pypy/-/blob/be829135bc0d758997b3566062999ee8b23872b4/lib-python/3/site.py#L250
23         return
24     warnings.warn(
25         "Distutils was imported before Setuptools, but importing Setuptools "
26         "also replaces the `distutils` module in `sys.modules`. This may lead "
27         "to undesirable behaviors or errors. To avoid these issues, avoid "
28         "using distutils directly, ensure that setuptools is installed in the "
29         "traditional way (e.g. not an editable install), and/or make sure "
30         "that setuptools is always imported before distutils.")
31 
32 
33 def clear_distutils():
34     if 'distutils' not in sys.modules:
35         return
36     warnings.warn("Setuptools is replacing distutils.")
37     mods = [name for name in sys.modules if re.match(r'distutils\b', name)]
38     for name in mods:
39         del sys.modules[name]
40 
41 
42 def enabled():
43     """
44     Allow selection of distutils by environment variable.
45     """
46     which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'local')
47     return which == 'local'
48 
49 
50 def ensure_local_distutils():
51     clear_distutils()
52 
53     # With the DistutilsMetaFinder in place,
54     # perform an import to cause distutils to be
55     # loaded from setuptools._distutils. Ref #2906.
56     with shim():
57         importlib.import_module('distutils')
58 
59     # check that submodules load as expected
60     core = importlib.import_module('distutils.core')
61     assert '_distutils' in core.__file__, core.__file__
62 
63 
64 def do_override():
65     """
66     Ensure that the local copy of distutils is preferred over stdlib.
67 
68     See https://github.com/pypa/setuptools/issues/417#issuecomment-392298401
69     for more motivation.
70     """
71     if enabled():
72         warn_distutils_present()
73         ensure_local_distutils()
74 
75 
76 class DistutilsMetaFinder:
77     def find_spec(self, fullname, path, target=None):
78         if path is not None:
79             return
80 
81         method_name = 'spec_for_{fullname}'.format(**locals())
82         method = getattr(self, method_name, lambda: None)
83         return method()
84 
85     def spec_for_distutils(self):
86         import importlib.abc
87         import importlib.util
88 
89         # In cases of path manipulation during sitecustomize,
90         # Setuptools might actually not be present even though
91         # the hook has been loaded. Allow the caller to fall
92         # back to stdlib behavior. See #2980.
93         if not importlib.util.find_spec('setuptools'):
94             return
95 
96         class DistutilsLoader(importlib.abc.Loader):
97 
98             def create_module(self, spec):
99                 return importlib.import_module('setuptools._distutils')
100 
101             def exec_module(self, module):
102                 pass
103 
104         return importlib.util.spec_from_loader('distutils', DistutilsLoader())
105 
106     def spec_for_pip(self):
107         """
108         Ensure stdlib distutils when running under pip.
109         See pypa/pip#8761 for rationale.
110         """
111         if self.pip_imported_during_build():
112             return
113         clear_distutils()
114         self.spec_for_distutils = lambda: None
115 
116     @classmethod
117     def pip_imported_during_build(cls):
118         """
119         Detect if pip is being imported in a build script. Ref #2355.
120         """
121         import traceback
122         return any(
123             cls.frame_file_is_setup(frame)
124             for frame, line in traceback.walk_stack(None)
125         )
126 
127     @staticmethod
128     def frame_file_is_setup(frame):
129         """
130         Return True if the indicated frame suggests a setup.py file.
131         """
132         # some frames may not have __file__ (#2940)
133         return frame.f_globals.get('__file__', '').endswith('setup.py')
134 
135 
136 DISTUTILS_FINDER = DistutilsMetaFinder()
137 
138 
139 def ensure_shim():
140     DISTUTILS_FINDER in sys.meta_path or add_shim()
141 
142 
143 @contextlib.contextmanager
144 def shim():
145     add_shim()
146     try:
147         yield
148     finally:
149         remove_shim()
150 
151 
152 def add_shim():
153     sys.meta_path.insert(0, DISTUTILS_FINDER)
154 
155 
156 def remove_shim():
157     try:
158         sys.meta_path.remove(DISTUTILS_FINDER)
159     except ValueError:
160         pass
161
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/_distutils_hack/__init__.py b/_distutils_hack/__init__.py
--- a/_distutils_hack/__init__.py
+++ b/_distutils_hack/__init__.py
@@ -86,17 +86,23 @@
         import importlib.abc
         import importlib.util
 
-        # In cases of path manipulation during sitecustomize,
-        # Setuptools might actually not be present even though
-        # the hook has been loaded. Allow the caller to fall
-        # back to stdlib behavior. See #2980.
-        if not importlib.util.find_spec('setuptools'):
+        try:
+            mod = importlib.import_module('setuptools._distutils')
+        except Exception:
+            # There are a couple of cases where setuptools._distutils
+            # may not be present:
+            # - An older Setuptools without a local distutils is
+            #   taking precedence. Ref #2957.
+            # - Path manipulation during sitecustomize removes
+            #   setuptools from the path but only after the hook
+            #   has been loaded. Ref #2980.
+            # In either case, fall back to stdlib behavior.
             return
 
         class DistutilsLoader(importlib.abc.Loader):
 
             def create_module(self, spec):
-                return importlib.import_module('setuptools._distutils')
+                return mod
 
             def exec_module(self, module):
                 pass

{"golden_diff": "diff --git a/_distutils_hack/__init__.py b/_distutils_hack/__init__.py\n--- a/_distutils_hack/__init__.py\n+++ b/_distutils_hack/__init__.py\n@@ -86,17 +86,23 @@\n         import importlib.abc\n         import importlib.util\n \n-        # In cases of path manipulation during sitecustomize,\n-        # Setuptools might actually not be present even though\n-        # the hook has been loaded. Allow the caller to fall\n-        # back to stdlib behavior. See #2980.\n-        if not importlib.util.find_spec('setuptools'):\n+        try:\n+            mod = importlib.import_module('setuptools._distutils')\n+        except Exception:\n+            # There are a couple of cases where setuptools._distutils\n+            # may not be present:\n+            # - An older Setuptools without a local distutils is\n+            #   taking precedence. Ref #2957.\n+            # - Path manipulation during sitecustomize removes\n+            #   setuptools from the path but only after the hook\n+            #   has been loaded. Ref #2980.\n+            # In either case, fall back to stdlib behavior.\n             return\n \n         class DistutilsLoader(importlib.abc.Loader):\n \n             def create_module(self, spec):\n-                return importlib.import_module('setuptools._distutils')\n+                return mod\n \n             def exec_module(self, module):\n                 pass\n", "issue": "[BUG] DistutilsMetaFinder breaks distutils imports when old setuptools is higher on path\n### setuptools version\r\n\r\nsetuptools==60.0.4\r\n\r\n### Python version\r\n\r\npython3.8\r\n\r\n### OS\r\n\r\nRHEL8, FreeBSD12/13, Ubuntu 20.04\r\n\r\n### Additional environment information\r\n\r\nAny environment where an old setuptools (eg OS package) is higher in `sys.path`, but a newer setuptools is installed anywhere on `sys.path` (with its distutils shimming `.pth` file active)\r\n\r\n### Description\r\n\r\nAny import of `distutils` will bomb when a new setuptools is present, but an older setuptools is higher on the path (since the new setuptools' path shim blindly tries to import from `setuptools._distutils`, which doesn't exist in old versions of setuptools). This has probably been the case ever since the distutils shim was created, but now that it's active by default, it's a bigger problem.\r\n\r\nThe place we hit this was running `pip install` from a venv with `--system-site-packages`- pip's build isolation installs latest `setuptools`, but then shuffles `sys.path` around such that the ancient OS-packaged `setuptools` is higher on the path. The first thing that tries to import `distutils` invokes the metapath finder, which has no validation or recovery and just lets the `ModuleNotFoundError` fly.\r\n\r\nAt first blush, it seems like the metapath shim's `spec_for_distutils` should try to at least verify the co-location of `setuptools` with the shim impl (ie, that they're under the same path prefix) and/or that the `_distutils` subpackage is available via `find_spec` before it actually imports it. Mixing and matching the shim version with random setuptools versions seems like a recipe for trouble down the road...\r\n\r\nI could probably throw together a PR for this if there's a consensus that this approach is sane.\r\n\r\n### Expected behavior\r\n\r\nThe presence of a new-but-not-loadable setuptools on sys.path doesn't break old setuptools/distutils.\r\n\r\n\r\n### How to Reproduce\r\n\r\n(NB: this is a much simpler repro than the OS installed setuptools that's actually failing, but illustrates the problem more easily on any OS)\r\n```\r\npython3 -m venv /tmp/bang && source /tmp/bang/bin/activate  # give us a clean venv to work in\r\npython -m pip install 'setuptools>60'  # install new setuptools to venv site-packages\r\npython -m pip install 'setuptools<45' -t .  # install old setuptools to cwd- higher than site-packages on path for most systems\r\npython -c 'import distutils'\r\n```\r\n\r\n### Output\r\n\r\n```console\r\n...\r\nTraceback (most recent call last):\r\n  File \"<string>\", line 1, in <module>\r\n  File \"/tmp/bang/lib64/python3.9/site-packages/_distutils_hack/__init__.py\", line 92, in create_module\r\n    return importlib.import_module('setuptools._distutils')\r\n  File \"/usr/lib64/python3.9/importlib/__init__.py\", line 127, in import_module\r\n    return _bootstrap._gcd_import(name[level:], package, level)\r\n  File \"/home/mdavis/setuptools/__init__.py\", line 6, in <module>\r\n    import distutils.core\r\n  File \"/tmp/bang/lib64/python3.9/site-packages/_distutils_hack/__init__.py\", line 92, in create_module\r\n    return importlib.import_module('setuptools._distutils')\r\n  File \"/usr/lib64/python3.9/importlib/__init__.py\", line 127, in import_module\r\n    return _bootstrap._gcd_import(name[level:], package, level)\r\nModuleNotFoundError: No module named 'setuptools._distutils'\r\n\r\n```\r\n\r\n\r\n### Code of Conduct\r\n\r\n- [X] I agree to follow the PSF Code of Conduct\n", "before_files": [{"content": "import sys\nimport os\nimport re\nimport importlib\nimport warnings\nimport contextlib\n\n\nis_pypy = '__pypy__' in sys.builtin_module_names\n\n\nwarnings.filterwarnings('ignore',\n                        r'.+ distutils\\b.+ deprecated',\n                        DeprecationWarning)\n\n\ndef warn_distutils_present():\n    if 'distutils' not in sys.modules:\n        return\n    if is_pypy and sys.version_info < (3, 7):\n        # PyPy for 3.6 unconditionally imports distutils, so bypass the warning\n        # https://foss.heptapod.net/pypy/pypy/-/blob/be829135bc0d758997b3566062999ee8b23872b4/lib-python/3/site.py#L250\n        return\n    warnings.warn(\n        \"Distutils was imported before Setuptools, but importing Setuptools \"\n        \"also replaces the `distutils` module in `sys.modules`. This may lead \"\n        \"to undesirable behaviors or errors. To avoid these issues, avoid \"\n        \"using distutils directly, ensure that setuptools is installed in the \"\n        \"traditional way (e.g. not an editable install), and/or make sure \"\n        \"that setuptools is always imported before distutils.\")\n\n\ndef clear_distutils():\n    if 'distutils' not in sys.modules:\n        return\n    warnings.warn(\"Setuptools is replacing distutils.\")\n    mods = [name for name in sys.modules if re.match(r'distutils\\b', name)]\n    for name in mods:\n        del sys.modules[name]\n\n\ndef enabled():\n    \"\"\"\n    Allow selection of distutils by environment variable.\n    \"\"\"\n    which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'local')\n    return which == 'local'\n\n\ndef ensure_local_distutils():\n    clear_distutils()\n\n    # With the DistutilsMetaFinder in place,\n    # perform an import to cause distutils to be\n    # loaded from setuptools._distutils. Ref #2906.\n    with shim():\n        importlib.import_module('distutils')\n\n    # check that submodules load as expected\n    core = importlib.import_module('distutils.core')\n    assert '_distutils' in core.__file__, core.__file__\n\n\ndef do_override():\n    \"\"\"\n    Ensure that the local copy of distutils is preferred over stdlib.\n\n    See https://github.com/pypa/setuptools/issues/417#issuecomment-392298401\n    for more motivation.\n    \"\"\"\n    if enabled():\n        warn_distutils_present()\n        ensure_local_distutils()\n\n\nclass DistutilsMetaFinder:\n    def find_spec(self, fullname, path, target=None):\n        if path is not None:\n            return\n\n        method_name = 'spec_for_{fullname}'.format(**locals())\n        method = getattr(self, method_name, lambda: None)\n        return method()\n\n    def spec_for_distutils(self):\n        import importlib.abc\n        import importlib.util\n\n        # In cases of path manipulation during sitecustomize,\n        # Setuptools might actually not be present even though\n        # the hook has been loaded. Allow the caller to fall\n        # back to stdlib behavior. See #2980.\n        if not importlib.util.find_spec('setuptools'):\n            return\n\n        class DistutilsLoader(importlib.abc.Loader):\n\n            def create_module(self, spec):\n                return importlib.import_module('setuptools._distutils')\n\n            def exec_module(self, module):\n                pass\n\n        return importlib.util.spec_from_loader('distutils', DistutilsLoader())\n\n    def spec_for_pip(self):\n        \"\"\"\n        Ensure stdlib distutils when running under pip.\n        See pypa/pip#8761 for rationale.\n        \"\"\"\n        if self.pip_imported_during_build():\n            return\n        clear_distutils()\n        self.spec_for_distutils = lambda: None\n\n    @classmethod\n    def pip_imported_during_build(cls):\n        \"\"\"\n        Detect if pip is being imported in a build script. Ref #2355.\n        \"\"\"\n        import traceback\n        return any(\n            cls.frame_file_is_setup(frame)\n            for frame, line in traceback.walk_stack(None)\n        )\n\n    @staticmethod\n    def frame_file_is_setup(frame):\n        \"\"\"\n        Return True if the indicated frame suggests a setup.py file.\n        \"\"\"\n        # some frames may not have __file__ (#2940)\n        return frame.f_globals.get('__file__', '').endswith('setup.py')\n\n\nDISTUTILS_FINDER = DistutilsMetaFinder()\n\n\ndef ensure_shim():\n    DISTUTILS_FINDER in sys.meta_path or add_shim()\n\n\n@contextlib.contextmanager\ndef shim():\n    add_shim()\n    try:\n        yield\n    finally:\n        remove_shim()\n\n\ndef add_shim():\n    sys.meta_path.insert(0, DISTUTILS_FINDER)\n\n\ndef remove_shim():\n    try:\n        sys.meta_path.remove(DISTUTILS_FINDER)\n    except ValueError:\n        pass\n", "path": "_distutils_hack/__init__.py"}], "after_files": [{"content": "import sys\nimport os\nimport re\nimport importlib\nimport warnings\nimport contextlib\n\n\nis_pypy = '__pypy__' in sys.builtin_module_names\n\n\nwarnings.filterwarnings('ignore',\n                        r'.+ distutils\\b.+ deprecated',\n                        DeprecationWarning)\n\n\ndef warn_distutils_present():\n    if 'distutils' not in sys.modules:\n        return\n    if is_pypy and sys.version_info < (3, 7):\n        # PyPy for 3.6 unconditionally imports distutils, so bypass the warning\n        # https://foss.heptapod.net/pypy/pypy/-/blob/be829135bc0d758997b3566062999ee8b23872b4/lib-python/3/site.py#L250\n        return\n    warnings.warn(\n        \"Distutils was imported before Setuptools, but importing Setuptools \"\n        \"also replaces the `distutils` module in `sys.modules`. This may lead \"\n        \"to undesirable behaviors or errors. To avoid these issues, avoid \"\n        \"using distutils directly, ensure that setuptools is installed in the \"\n        \"traditional way (e.g. not an editable install), and/or make sure \"\n        \"that setuptools is always imported before distutils.\")\n\n\ndef clear_distutils():\n    if 'distutils' not in sys.modules:\n        return\n    warnings.warn(\"Setuptools is replacing distutils.\")\n    mods = [name for name in sys.modules if re.match(r'distutils\\b', name)]\n    for name in mods:\n        del sys.modules[name]\n\n\ndef enabled():\n    \"\"\"\n    Allow selection of distutils by environment variable.\n    \"\"\"\n    which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'local')\n    return which == 'local'\n\n\ndef ensure_local_distutils():\n    clear_distutils()\n\n    # With the DistutilsMetaFinder in place,\n    # perform an import to cause distutils to be\n    # loaded from setuptools._distutils. Ref #2906.\n    with shim():\n        importlib.import_module('distutils')\n\n    # check that submodules load as expected\n    core = importlib.import_module('distutils.core')\n    assert '_distutils' in core.__file__, core.__file__\n\n\ndef do_override():\n    \"\"\"\n    Ensure that the local copy of distutils is preferred over stdlib.\n\n    See https://github.com/pypa/setuptools/issues/417#issuecomment-392298401\n    for more motivation.\n    \"\"\"\n    if enabled():\n        warn_distutils_present()\n        ensure_local_distutils()\n\n\nclass DistutilsMetaFinder:\n    def find_spec(self, fullname, path, target=None):\n        if path is not None:\n            return\n\n        method_name = 'spec_for_{fullname}'.format(**locals())\n        method = getattr(self, method_name, lambda: None)\n        return method()\n\n    def spec_for_distutils(self):\n        import importlib.abc\n        import importlib.util\n\n        try:\n            mod = importlib.import_module('setuptools._distutils')\n        except Exception:\n            # There are a couple of cases where setuptools._distutils\n            # may not be present:\n            # - An older Setuptools without a local distutils is\n            #   taking precedence. Ref #2957.\n            # - Path manipulation during sitecustomize removes\n            #   setuptools from the path but only after the hook\n            #   has been loaded. Ref #2980.\n            # In either case, fall back to stdlib behavior.\n            return\n\n        class DistutilsLoader(importlib.abc.Loader):\n\n            def create_module(self, spec):\n                return mod\n\n            def exec_module(self, module):\n                pass\n\n        return importlib.util.spec_from_loader('distutils', DistutilsLoader())\n\n    def spec_for_pip(self):\n        \"\"\"\n        Ensure stdlib distutils when running under pip.\n        See pypa/pip#8761 for rationale.\n        \"\"\"\n        if self.pip_imported_during_build():\n            return\n        clear_distutils()\n        self.spec_for_distutils = lambda: None\n\n    @classmethod\n    def pip_imported_during_build(cls):\n        \"\"\"\n        Detect if pip is being imported in a build script. Ref #2355.\n        \"\"\"\n        import traceback\n        return any(\n            cls.frame_file_is_setup(frame)\n            for frame, line in traceback.walk_stack(None)\n        )\n\n    @staticmethod\n    def frame_file_is_setup(frame):\n        \"\"\"\n        Return True if the indicated frame suggests a setup.py file.\n        \"\"\"\n        # some frames may not have __file__ (#2940)\n        return frame.f_globals.get('__file__', '').endswith('setup.py')\n\n\nDISTUTILS_FINDER = DistutilsMetaFinder()\n\n\ndef ensure_shim():\n    DISTUTILS_FINDER in sys.meta_path or add_shim()\n\n\n@contextlib.contextmanager\ndef shim():\n    add_shim()\n    try:\n        yield\n    finally:\n        remove_shim()\n\n\ndef add_shim():\n    sys.meta_path.insert(0, DISTUTILS_FINDER)\n\n\ndef remove_shim():\n    try:\n        sys.meta_path.remove(DISTUTILS_FINDER)\n    except ValueError:\n        pass\n", "path": "_distutils_hack/__init__.py"}]}