We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Let black run on files even if `ast.parse` with `type_comments=True` fails
**Is your feature request related to a problem? Please describe.**

Right now, if a file parsing because of invalid type comments (even if the code runs just fine), black won't format the file.

For example:

```python
import typing

def foo(
    # type: typing.List
    bar,
):
    print(bar)

foo(10)
```

While the code still runs:

```console
$ python3 asd.py            
10
```

Black no longer formats it:

```console
$ black asd.py        
error: cannot format asd.py: cannot use --safe with this file; failed to parse source file AST: invalid syntax (<unknown>, line 4)
This could be caused by running Black with an older Python version that does not support new syntax used in your source file.

Oh no! 💥 💔 💥
1 file failed to reformat.
```

This is because `ast.parse` on that code fails if you pass `type_comments=True`:

```pycon
>>> import ast
>>> ast.parse(open('asd.py').read(), type_comments=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 4
    # type: typing.List
            ^^^^^^^^^^^
SyntaxError: invalid syntax
```

(Black does work if you do `black asd.py --fast`, as it no longer does the AST based check.)

**Describe the solution you'd like**

If `ast.parse(..., type_comments=True)` fails for the before/after check, maybe doing it with `type_comments=False` is fine, just for that file?

**Describe alternatives you've considered**

None so far.

**Additional context**

I don't think this can be considered an upstream bug, as this is indeed invalid syntax for a type checker's perspective. But black shouldn't care if the types are correct, in any way shape or form.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/black/parsing.py`
Content:
```
1 """
2 Parse Python code and perform AST validation.
3 """
4 import ast
5 import platform
6 import sys
7 from typing import Any, Iterable, Iterator, List, Set, Tuple, Type, Union
8 
9 if sys.version_info < (3, 8):
10     from typing_extensions import Final
11 else:
12     from typing import Final
13 
14 from black.mode import VERSION_TO_FEATURES, Feature, TargetVersion, supports_feature
15 from black.nodes import syms
16 from blib2to3 import pygram
17 from blib2to3.pgen2 import driver
18 from blib2to3.pgen2.grammar import Grammar
19 from blib2to3.pgen2.parse import ParseError
20 from blib2to3.pgen2.tokenize import TokenError
21 from blib2to3.pytree import Leaf, Node
22 
23 ast3: Any
24 
25 _IS_PYPY = platform.python_implementation() == "PyPy"
26 
27 try:
28     from typed_ast import ast3
29 except ImportError:
30     if sys.version_info < (3, 8) and not _IS_PYPY:
31         print(
32             (
33                 "The typed_ast package is required but not installed.\n"
34                 "You can upgrade to Python 3.8+ or install typed_ast with\n"
35                 "`python3 -m pip install typed-ast`."
36             ),
37             file=sys.stderr,
38         )
39         sys.exit(1)
40     else:
41         ast3 = ast
42 
43 
44 PY2_HINT: Final = "Python 2 support was removed in version 22.0."
45 
46 
47 class InvalidInput(ValueError):
48     """Raised when input source code fails all parse attempts."""
49 
50 
51 def get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:
52     if not target_versions:
53         # No target_version specified, so try all grammars.
54         return [
55             # Python 3.7-3.9
56             pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords,
57             # Python 3.0-3.6
58             pygram.python_grammar_no_print_statement_no_exec_statement,
59             # Python 3.10+
60             pygram.python_grammar_soft_keywords,
61         ]
62 
63     grammars = []
64     # If we have to parse both, try to parse async as a keyword first
65     if not supports_feature(
66         target_versions, Feature.ASYNC_IDENTIFIERS
67     ) and not supports_feature(target_versions, Feature.PATTERN_MATCHING):
68         # Python 3.7-3.9
69         grammars.append(
70             pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords
71         )
72     if not supports_feature(target_versions, Feature.ASYNC_KEYWORDS):
73         # Python 3.0-3.6
74         grammars.append(pygram.python_grammar_no_print_statement_no_exec_statement)
75     if any(Feature.PATTERN_MATCHING in VERSION_TO_FEATURES[v] for v in target_versions):
76         # Python 3.10+
77         grammars.append(pygram.python_grammar_soft_keywords)
78 
79     # At least one of the above branches must have been taken, because every Python
80     # version has exactly one of the two 'ASYNC_*' flags
81     return grammars
82 
83 
84 def lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:
85     """Given a string with source, return the lib2to3 Node."""
86     if not src_txt.endswith("\n"):
87         src_txt += "\n"
88 
89     grammars = get_grammars(set(target_versions))
90     errors = {}
91     for grammar in grammars:
92         drv = driver.Driver(grammar)
93         try:
94             result = drv.parse_string(src_txt, True)
95             break
96 
97         except ParseError as pe:
98             lineno, column = pe.context[1]
99             lines = src_txt.splitlines()
100             try:
101                 faulty_line = lines[lineno - 1]
102             except IndexError:
103                 faulty_line = "<line number missing in source>"
104             errors[grammar.version] = InvalidInput(
105                 f"Cannot parse: {lineno}:{column}: {faulty_line}"
106             )
107 
108         except TokenError as te:
109             # In edge cases these are raised; and typically don't have a "faulty_line".
110             lineno, column = te.args[1]
111             errors[grammar.version] = InvalidInput(
112                 f"Cannot parse: {lineno}:{column}: {te.args[0]}"
113             )
114 
115     else:
116         # Choose the latest version when raising the actual parsing error.
117         assert len(errors) >= 1
118         exc = errors[max(errors)]
119 
120         if matches_grammar(src_txt, pygram.python_grammar) or matches_grammar(
121             src_txt, pygram.python_grammar_no_print_statement
122         ):
123             original_msg = exc.args[0]
124             msg = f"{original_msg}\n{PY2_HINT}"
125             raise InvalidInput(msg) from None
126 
127         raise exc from None
128 
129     if isinstance(result, Leaf):
130         result = Node(syms.file_input, [result])
131     return result
132 
133 
134 def matches_grammar(src_txt: str, grammar: Grammar) -> bool:
135     drv = driver.Driver(grammar)
136     try:
137         drv.parse_string(src_txt, True)
138     except (ParseError, TokenError, IndentationError):
139         return False
140     else:
141         return True
142 
143 
144 def lib2to3_unparse(node: Node) -> str:
145     """Given a lib2to3 node, return its string representation."""
146     code = str(node)
147     return code
148 
149 
150 def parse_single_version(
151     src: str, version: Tuple[int, int]
152 ) -> Union[ast.AST, ast3.AST]:
153     filename = "<unknown>"
154     # typed-ast is needed because of feature version limitations in the builtin ast 3.8>
155     if sys.version_info >= (3, 8) and version >= (3,):
156         return ast.parse(src, filename, feature_version=version, type_comments=True)
157 
158     if _IS_PYPY:
159         # PyPy 3.7 doesn't support type comment tracking which is not ideal, but there's
160         # not much we can do as typed-ast won't work either.
161         if sys.version_info >= (3, 8):
162             return ast3.parse(src, filename, type_comments=True)
163         else:
164             return ast3.parse(src, filename)
165     else:
166         # Typed-ast is guaranteed to be used here and automatically tracks type
167         # comments separately.
168         return ast3.parse(src, filename, feature_version=version[1])
169 
170 
171 def parse_ast(src: str) -> Union[ast.AST, ast3.AST]:
172     # TODO: support Python 4+ ;)
173     versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]
174 
175     first_error = ""
176     for version in sorted(versions, reverse=True):
177         try:
178             return parse_single_version(src, version)
179         except SyntaxError as e:
180             if not first_error:
181                 first_error = str(e)
182 
183     raise SyntaxError(first_error)
184 
185 
186 ast3_AST: Final[Type[ast3.AST]] = ast3.AST
187 
188 
189 def _normalize(lineend: str, value: str) -> str:
190     # To normalize, we strip any leading and trailing space from
191     # each line...
192     stripped: List[str] = [i.strip() for i in value.splitlines()]
193     normalized = lineend.join(stripped)
194     # ...and remove any blank lines at the beginning and end of
195     # the whole string
196     return normalized.strip()
197 
198 
199 def stringify_ast(node: Union[ast.AST, ast3.AST], depth: int = 0) -> Iterator[str]:
200     """Simple visitor generating strings to compare ASTs by content."""
201 
202     node = fixup_ast_constants(node)
203 
204     yield f"{'  ' * depth}{node.__class__.__name__}("
205 
206     type_ignore_classes: Tuple[Type[Any], ...]
207     for field in sorted(node._fields):  # noqa: F402
208         # TypeIgnore will not be present using pypy < 3.8, so need for this
209         if not (_IS_PYPY and sys.version_info < (3, 8)):
210             # TypeIgnore has only one field 'lineno' which breaks this comparison
211             type_ignore_classes = (ast3.TypeIgnore,)
212             if sys.version_info >= (3, 8):
213                 type_ignore_classes += (ast.TypeIgnore,)
214             if isinstance(node, type_ignore_classes):
215                 break
216 
217         try:
218             value: object = getattr(node, field)
219         except AttributeError:
220             continue
221 
222         yield f"{'  ' * (depth+1)}{field}="
223 
224         if isinstance(value, list):
225             for item in value:
226                 # Ignore nested tuples within del statements, because we may insert
227                 # parentheses and they change the AST.
228                 if (
229                     field == "targets"
230                     and isinstance(node, (ast.Delete, ast3.Delete))
231                     and isinstance(item, (ast.Tuple, ast3.Tuple))
232                 ):
233                     for elt in item.elts:
234                         yield from stringify_ast(elt, depth + 2)
235 
236                 elif isinstance(item, (ast.AST, ast3.AST)):
237                     yield from stringify_ast(item, depth + 2)
238 
239         # Note that we are referencing the typed-ast ASTs via global variables and not
240         # direct module attribute accesses because that breaks mypyc. It's probably
241         # something to do with the ast3 variables being marked as Any leading
242         # mypy to think this branch is always taken, leaving the rest of the code
243         # unanalyzed. Tighting up the types for the typed-ast AST types avoids the
244         # mypyc crash.
245         elif isinstance(value, (ast.AST, ast3_AST)):
246             yield from stringify_ast(value, depth + 2)
247 
248         else:
249             normalized: object
250             # Constant strings may be indented across newlines, if they are
251             # docstrings; fold spaces after newlines when comparing. Similarly,
252             # trailing and leading space may be removed.
253             if (
254                 isinstance(node, ast.Constant)
255                 and field == "value"
256                 and isinstance(value, str)
257             ):
258                 normalized = _normalize("\n", value)
259             else:
260                 normalized = value
261             yield f"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}"
262 
263     yield f"{'  ' * depth})  # /{node.__class__.__name__}"
264 
265 
266 def fixup_ast_constants(node: Union[ast.AST, ast3.AST]) -> Union[ast.AST, ast3.AST]:
267     """Map ast nodes deprecated in 3.8 to Constant."""
268     if isinstance(node, (ast.Str, ast3.Str, ast.Bytes, ast3.Bytes)):
269         return ast.Constant(value=node.s)
270 
271     if isinstance(node, (ast.Num, ast3.Num)):
272         return ast.Constant(value=node.n)
273 
274     if isinstance(node, (ast.NameConstant, ast3.NameConstant)):
275         return ast.Constant(value=node.value)
276 
277     return node
278
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/black/parsing.py b/src/black/parsing.py
--- a/src/black/parsing.py
+++ b/src/black/parsing.py
@@ -148,24 +148,29 @@
 
 
 def parse_single_version(
-    src: str, version: Tuple[int, int]
+    src: str, version: Tuple[int, int], *, type_comments: bool
 ) -> Union[ast.AST, ast3.AST]:
     filename = "<unknown>"
     # typed-ast is needed because of feature version limitations in the builtin ast 3.8>
     if sys.version_info >= (3, 8) and version >= (3,):
-        return ast.parse(src, filename, feature_version=version, type_comments=True)
+        return ast.parse(
+            src, filename, feature_version=version, type_comments=type_comments
+        )
 
     if _IS_PYPY:
         # PyPy 3.7 doesn't support type comment tracking which is not ideal, but there's
         # not much we can do as typed-ast won't work either.
         if sys.version_info >= (3, 8):
-            return ast3.parse(src, filename, type_comments=True)
+            return ast3.parse(src, filename, type_comments=type_comments)
         else:
             return ast3.parse(src, filename)
     else:
-        # Typed-ast is guaranteed to be used here and automatically tracks type
-        # comments separately.
-        return ast3.parse(src, filename, feature_version=version[1])
+        if type_comments:
+            # Typed-ast is guaranteed to be used here and automatically tracks type
+            # comments separately.
+            return ast3.parse(src, filename, feature_version=version[1])
+        else:
+            return ast.parse(src, filename)
 
 
 def parse_ast(src: str) -> Union[ast.AST, ast3.AST]:
@@ -175,11 +180,18 @@
     first_error = ""
     for version in sorted(versions, reverse=True):
         try:
-            return parse_single_version(src, version)
+            return parse_single_version(src, version, type_comments=True)
         except SyntaxError as e:
             if not first_error:
                 first_error = str(e)
 
+    # Try to parse without type comments
+    for version in sorted(versions, reverse=True):
+        try:
+            return parse_single_version(src, version, type_comments=False)
+        except SyntaxError:
+            pass
+
     raise SyntaxError(first_error)

{"golden_diff": "diff --git a/src/black/parsing.py b/src/black/parsing.py\n--- a/src/black/parsing.py\n+++ b/src/black/parsing.py\n@@ -148,24 +148,29 @@\n \n \n def parse_single_version(\n-    src: str, version: Tuple[int, int]\n+    src: str, version: Tuple[int, int], *, type_comments: bool\n ) -> Union[ast.AST, ast3.AST]:\n     filename = \"<unknown>\"\n     # typed-ast is needed because of feature version limitations in the builtin ast 3.8>\n     if sys.version_info >= (3, 8) and version >= (3,):\n-        return ast.parse(src, filename, feature_version=version, type_comments=True)\n+        return ast.parse(\n+            src, filename, feature_version=version, type_comments=type_comments\n+        )\n \n     if _IS_PYPY:\n         # PyPy 3.7 doesn't support type comment tracking which is not ideal, but there's\n         # not much we can do as typed-ast won't work either.\n         if sys.version_info >= (3, 8):\n-            return ast3.parse(src, filename, type_comments=True)\n+            return ast3.parse(src, filename, type_comments=type_comments)\n         else:\n             return ast3.parse(src, filename)\n     else:\n-        # Typed-ast is guaranteed to be used here and automatically tracks type\n-        # comments separately.\n-        return ast3.parse(src, filename, feature_version=version[1])\n+        if type_comments:\n+            # Typed-ast is guaranteed to be used here and automatically tracks type\n+            # comments separately.\n+            return ast3.parse(src, filename, feature_version=version[1])\n+        else:\n+            return ast.parse(src, filename)\n \n \n def parse_ast(src: str) -> Union[ast.AST, ast3.AST]:\n@@ -175,11 +180,18 @@\n     first_error = \"\"\n     for version in sorted(versions, reverse=True):\n         try:\n-            return parse_single_version(src, version)\n+            return parse_single_version(src, version, type_comments=True)\n         except SyntaxError as e:\n             if not first_error:\n                 first_error = str(e)\n \n+    # Try to parse without type comments\n+    for version in sorted(versions, reverse=True):\n+        try:\n+            return parse_single_version(src, version, type_comments=False)\n+        except SyntaxError:\n+            pass\n+\n     raise SyntaxError(first_error)\n", "issue": "Let black run on files even if `ast.parse` with `type_comments=True` fails\n**Is your feature request related to a problem? Please describe.**\r\n\r\nRight now, if a file parsing because of invalid type comments (even if the code runs just fine), black won't format the file.\r\n\r\nFor example:\r\n\r\n```python\r\nimport typing\r\n\r\ndef foo(\r\n    # type: typing.List\r\n    bar,\r\n):\r\n    print(bar)\r\n\r\nfoo(10)\r\n```\r\n\r\nWhile the code still runs:\r\n\r\n```console\r\n$ python3 asd.py            \r\n10\r\n```\r\n\r\nBlack no longer formats it:\r\n\r\n```console\r\n$ black asd.py        \r\nerror: cannot format asd.py: cannot use --safe with this file; failed to parse source file AST: invalid syntax (<unknown>, line 4)\r\nThis could be caused by running Black with an older Python version that does not support new syntax used in your source file.\r\n\r\nOh no! \ud83d\udca5 \ud83d\udc94 \ud83d\udca5\r\n1 file failed to reformat.\r\n```\r\n\r\nThis is because `ast.parse` on that code fails if you pass `type_comments=True`:\r\n\r\n```pycon\r\n>>> import ast\r\n>>> ast.parse(open('asd.py').read(), type_comments=True)\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py\", line 50, in parse\r\n    return compile(source, filename, mode, flags,\r\n  File \"<unknown>\", line 4\r\n    # type: typing.List\r\n            ^^^^^^^^^^^\r\nSyntaxError: invalid syntax\r\n```\r\n\r\n(Black does work if you do `black asd.py --fast`, as it no longer does the AST based check.)\r\n\r\n**Describe the solution you'd like**\r\n\r\nIf `ast.parse(..., type_comments=True)` fails for the before/after check, maybe doing it with `type_comments=False` is fine, just for that file?\r\n\r\n**Describe alternatives you've considered**\r\n\r\nNone so far.\r\n\r\n**Additional context**\r\n\r\nI don't think this can be considered an upstream bug, as this is indeed invalid syntax for a type checker's perspective. But black shouldn't care if the types are correct, in any way shape or form.\r\n\n", "before_files": [{"content": "\"\"\"\nParse Python code and perform AST validation.\n\"\"\"\nimport ast\nimport platform\nimport sys\nfrom typing import Any, Iterable, Iterator, List, Set, Tuple, Type, Union\n\nif sys.version_info < (3, 8):\n    from typing_extensions import Final\nelse:\n    from typing import Final\n\nfrom black.mode import VERSION_TO_FEATURES, Feature, TargetVersion, supports_feature\nfrom black.nodes import syms\nfrom blib2to3 import pygram\nfrom blib2to3.pgen2 import driver\nfrom blib2to3.pgen2.grammar import Grammar\nfrom blib2to3.pgen2.parse import ParseError\nfrom blib2to3.pgen2.tokenize import TokenError\nfrom blib2to3.pytree import Leaf, Node\n\nast3: Any\n\n_IS_PYPY = platform.python_implementation() == \"PyPy\"\n\ntry:\n    from typed_ast import ast3\nexcept ImportError:\n    if sys.version_info < (3, 8) and not _IS_PYPY:\n        print(\n            (\n                \"The typed_ast package is required but not installed.\\n\"\n                \"You can upgrade to Python 3.8+ or install typed_ast with\\n\"\n                \"`python3 -m pip install typed-ast`.\"\n            ),\n            file=sys.stderr,\n        )\n        sys.exit(1)\n    else:\n        ast3 = ast\n\n\nPY2_HINT: Final = \"Python 2 support was removed in version 22.0.\"\n\n\nclass InvalidInput(ValueError):\n    \"\"\"Raised when input source code fails all parse attempts.\"\"\"\n\n\ndef get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:\n    if not target_versions:\n        # No target_version specified, so try all grammars.\n        return [\n            # Python 3.7-3.9\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords,\n            # Python 3.0-3.6\n            pygram.python_grammar_no_print_statement_no_exec_statement,\n            # Python 3.10+\n            pygram.python_grammar_soft_keywords,\n        ]\n\n    grammars = []\n    # If we have to parse both, try to parse async as a keyword first\n    if not supports_feature(\n        target_versions, Feature.ASYNC_IDENTIFIERS\n    ) and not supports_feature(target_versions, Feature.PATTERN_MATCHING):\n        # Python 3.7-3.9\n        grammars.append(\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords\n        )\n    if not supports_feature(target_versions, Feature.ASYNC_KEYWORDS):\n        # Python 3.0-3.6\n        grammars.append(pygram.python_grammar_no_print_statement_no_exec_statement)\n    if any(Feature.PATTERN_MATCHING in VERSION_TO_FEATURES[v] for v in target_versions):\n        # Python 3.10+\n        grammars.append(pygram.python_grammar_soft_keywords)\n\n    # At least one of the above branches must have been taken, because every Python\n    # version has exactly one of the two 'ASYNC_*' flags\n    return grammars\n\n\ndef lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:\n    \"\"\"Given a string with source, return the lib2to3 Node.\"\"\"\n    if not src_txt.endswith(\"\\n\"):\n        src_txt += \"\\n\"\n\n    grammars = get_grammars(set(target_versions))\n    errors = {}\n    for grammar in grammars:\n        drv = driver.Driver(grammar)\n        try:\n            result = drv.parse_string(src_txt, True)\n            break\n\n        except ParseError as pe:\n            lineno, column = pe.context[1]\n            lines = src_txt.splitlines()\n            try:\n                faulty_line = lines[lineno - 1]\n            except IndexError:\n                faulty_line = \"<line number missing in source>\"\n            errors[grammar.version] = InvalidInput(\n                f\"Cannot parse: {lineno}:{column}: {faulty_line}\"\n            )\n\n        except TokenError as te:\n            # In edge cases these are raised; and typically don't have a \"faulty_line\".\n            lineno, column = te.args[1]\n            errors[grammar.version] = InvalidInput(\n                f\"Cannot parse: {lineno}:{column}: {te.args[0]}\"\n            )\n\n    else:\n        # Choose the latest version when raising the actual parsing error.\n        assert len(errors) >= 1\n        exc = errors[max(errors)]\n\n        if matches_grammar(src_txt, pygram.python_grammar) or matches_grammar(\n            src_txt, pygram.python_grammar_no_print_statement\n        ):\n            original_msg = exc.args[0]\n            msg = f\"{original_msg}\\n{PY2_HINT}\"\n            raise InvalidInput(msg) from None\n\n        raise exc from None\n\n    if isinstance(result, Leaf):\n        result = Node(syms.file_input, [result])\n    return result\n\n\ndef matches_grammar(src_txt: str, grammar: Grammar) -> bool:\n    drv = driver.Driver(grammar)\n    try:\n        drv.parse_string(src_txt, True)\n    except (ParseError, TokenError, IndentationError):\n        return False\n    else:\n        return True\n\n\ndef lib2to3_unparse(node: Node) -> str:\n    \"\"\"Given a lib2to3 node, return its string representation.\"\"\"\n    code = str(node)\n    return code\n\n\ndef parse_single_version(\n    src: str, version: Tuple[int, int]\n) -> Union[ast.AST, ast3.AST]:\n    filename = \"<unknown>\"\n    # typed-ast is needed because of feature version limitations in the builtin ast 3.8>\n    if sys.version_info >= (3, 8) and version >= (3,):\n        return ast.parse(src, filename, feature_version=version, type_comments=True)\n\n    if _IS_PYPY:\n        # PyPy 3.7 doesn't support type comment tracking which is not ideal, but there's\n        # not much we can do as typed-ast won't work either.\n        if sys.version_info >= (3, 8):\n            return ast3.parse(src, filename, type_comments=True)\n        else:\n            return ast3.parse(src, filename)\n    else:\n        # Typed-ast is guaranteed to be used here and automatically tracks type\n        # comments separately.\n        return ast3.parse(src, filename, feature_version=version[1])\n\n\ndef parse_ast(src: str) -> Union[ast.AST, ast3.AST]:\n    # TODO: support Python 4+ ;)\n    versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]\n\n    first_error = \"\"\n    for version in sorted(versions, reverse=True):\n        try:\n            return parse_single_version(src, version)\n        except SyntaxError as e:\n            if not first_error:\n                first_error = str(e)\n\n    raise SyntaxError(first_error)\n\n\nast3_AST: Final[Type[ast3.AST]] = ast3.AST\n\n\ndef _normalize(lineend: str, value: str) -> str:\n    # To normalize, we strip any leading and trailing space from\n    # each line...\n    stripped: List[str] = [i.strip() for i in value.splitlines()]\n    normalized = lineend.join(stripped)\n    # ...and remove any blank lines at the beginning and end of\n    # the whole string\n    return normalized.strip()\n\n\ndef stringify_ast(node: Union[ast.AST, ast3.AST], depth: int = 0) -> Iterator[str]:\n    \"\"\"Simple visitor generating strings to compare ASTs by content.\"\"\"\n\n    node = fixup_ast_constants(node)\n\n    yield f\"{'  ' * depth}{node.__class__.__name__}(\"\n\n    type_ignore_classes: Tuple[Type[Any], ...]\n    for field in sorted(node._fields):  # noqa: F402\n        # TypeIgnore will not be present using pypy < 3.8, so need for this\n        if not (_IS_PYPY and sys.version_info < (3, 8)):\n            # TypeIgnore has only one field 'lineno' which breaks this comparison\n            type_ignore_classes = (ast3.TypeIgnore,)\n            if sys.version_info >= (3, 8):\n                type_ignore_classes += (ast.TypeIgnore,)\n            if isinstance(node, type_ignore_classes):\n                break\n\n        try:\n            value: object = getattr(node, field)\n        except AttributeError:\n            continue\n\n        yield f\"{'  ' * (depth+1)}{field}=\"\n\n        if isinstance(value, list):\n            for item in value:\n                # Ignore nested tuples within del statements, because we may insert\n                # parentheses and they change the AST.\n                if (\n                    field == \"targets\"\n                    and isinstance(node, (ast.Delete, ast3.Delete))\n                    and isinstance(item, (ast.Tuple, ast3.Tuple))\n                ):\n                    for elt in item.elts:\n                        yield from stringify_ast(elt, depth + 2)\n\n                elif isinstance(item, (ast.AST, ast3.AST)):\n                    yield from stringify_ast(item, depth + 2)\n\n        # Note that we are referencing the typed-ast ASTs via global variables and not\n        # direct module attribute accesses because that breaks mypyc. It's probably\n        # something to do with the ast3 variables being marked as Any leading\n        # mypy to think this branch is always taken, leaving the rest of the code\n        # unanalyzed. Tighting up the types for the typed-ast AST types avoids the\n        # mypyc crash.\n        elif isinstance(value, (ast.AST, ast3_AST)):\n            yield from stringify_ast(value, depth + 2)\n\n        else:\n            normalized: object\n            # Constant strings may be indented across newlines, if they are\n            # docstrings; fold spaces after newlines when comparing. Similarly,\n            # trailing and leading space may be removed.\n            if (\n                isinstance(node, ast.Constant)\n                and field == \"value\"\n                and isinstance(value, str)\n            ):\n                normalized = _normalize(\"\\n\", value)\n            else:\n                normalized = value\n            yield f\"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}\"\n\n    yield f\"{'  ' * depth})  # /{node.__class__.__name__}\"\n\n\ndef fixup_ast_constants(node: Union[ast.AST, ast3.AST]) -> Union[ast.AST, ast3.AST]:\n    \"\"\"Map ast nodes deprecated in 3.8 to Constant.\"\"\"\n    if isinstance(node, (ast.Str, ast3.Str, ast.Bytes, ast3.Bytes)):\n        return ast.Constant(value=node.s)\n\n    if isinstance(node, (ast.Num, ast3.Num)):\n        return ast.Constant(value=node.n)\n\n    if isinstance(node, (ast.NameConstant, ast3.NameConstant)):\n        return ast.Constant(value=node.value)\n\n    return node\n", "path": "src/black/parsing.py"}], "after_files": [{"content": "\"\"\"\nParse Python code and perform AST validation.\n\"\"\"\nimport ast\nimport platform\nimport sys\nfrom typing import Any, Iterable, Iterator, List, Set, Tuple, Type, Union\n\nif sys.version_info < (3, 8):\n    from typing_extensions import Final\nelse:\n    from typing import Final\n\nfrom black.mode import VERSION_TO_FEATURES, Feature, TargetVersion, supports_feature\nfrom black.nodes import syms\nfrom blib2to3 import pygram\nfrom blib2to3.pgen2 import driver\nfrom blib2to3.pgen2.grammar import Grammar\nfrom blib2to3.pgen2.parse import ParseError\nfrom blib2to3.pgen2.tokenize import TokenError\nfrom blib2to3.pytree import Leaf, Node\n\nast3: Any\n\n_IS_PYPY = platform.python_implementation() == \"PyPy\"\n\ntry:\n    from typed_ast import ast3\nexcept ImportError:\n    if sys.version_info < (3, 8) and not _IS_PYPY:\n        print(\n            (\n                \"The typed_ast package is required but not installed.\\n\"\n                \"You can upgrade to Python 3.8+ or install typed_ast with\\n\"\n                \"`python3 -m pip install typed-ast`.\"\n            ),\n            file=sys.stderr,\n        )\n        sys.exit(1)\n    else:\n        ast3 = ast\n\n\nPY2_HINT: Final = \"Python 2 support was removed in version 22.0.\"\n\n\nclass InvalidInput(ValueError):\n    \"\"\"Raised when input source code fails all parse attempts.\"\"\"\n\n\ndef get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:\n    if not target_versions:\n        # No target_version specified, so try all grammars.\n        return [\n            # Python 3.7-3.9\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords,\n            # Python 3.0-3.6\n            pygram.python_grammar_no_print_statement_no_exec_statement,\n            # Python 3.10+\n            pygram.python_grammar_soft_keywords,\n        ]\n\n    grammars = []\n    # If we have to parse both, try to parse async as a keyword first\n    if not supports_feature(\n        target_versions, Feature.ASYNC_IDENTIFIERS\n    ) and not supports_feature(target_versions, Feature.PATTERN_MATCHING):\n        # Python 3.7-3.9\n        grammars.append(\n            pygram.python_grammar_no_print_statement_no_exec_statement_async_keywords\n        )\n    if not supports_feature(target_versions, Feature.ASYNC_KEYWORDS):\n        # Python 3.0-3.6\n        grammars.append(pygram.python_grammar_no_print_statement_no_exec_statement)\n    if any(Feature.PATTERN_MATCHING in VERSION_TO_FEATURES[v] for v in target_versions):\n        # Python 3.10+\n        grammars.append(pygram.python_grammar_soft_keywords)\n\n    # At least one of the above branches must have been taken, because every Python\n    # version has exactly one of the two 'ASYNC_*' flags\n    return grammars\n\n\ndef lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:\n    \"\"\"Given a string with source, return the lib2to3 Node.\"\"\"\n    if not src_txt.endswith(\"\\n\"):\n        src_txt += \"\\n\"\n\n    grammars = get_grammars(set(target_versions))\n    errors = {}\n    for grammar in grammars:\n        drv = driver.Driver(grammar)\n        try:\n            result = drv.parse_string(src_txt, True)\n            break\n\n        except ParseError as pe:\n            lineno, column = pe.context[1]\n            lines = src_txt.splitlines()\n            try:\n                faulty_line = lines[lineno - 1]\n            except IndexError:\n                faulty_line = \"<line number missing in source>\"\n            errors[grammar.version] = InvalidInput(\n                f\"Cannot parse: {lineno}:{column}: {faulty_line}\"\n            )\n\n        except TokenError as te:\n            # In edge cases these are raised; and typically don't have a \"faulty_line\".\n            lineno, column = te.args[1]\n            errors[grammar.version] = InvalidInput(\n                f\"Cannot parse: {lineno}:{column}: {te.args[0]}\"\n            )\n\n    else:\n        # Choose the latest version when raising the actual parsing error.\n        assert len(errors) >= 1\n        exc = errors[max(errors)]\n\n        if matches_grammar(src_txt, pygram.python_grammar) or matches_grammar(\n            src_txt, pygram.python_grammar_no_print_statement\n        ):\n            original_msg = exc.args[0]\n            msg = f\"{original_msg}\\n{PY2_HINT}\"\n            raise InvalidInput(msg) from None\n\n        raise exc from None\n\n    if isinstance(result, Leaf):\n        result = Node(syms.file_input, [result])\n    return result\n\n\ndef matches_grammar(src_txt: str, grammar: Grammar) -> bool:\n    drv = driver.Driver(grammar)\n    try:\n        drv.parse_string(src_txt, True)\n    except (ParseError, TokenError, IndentationError):\n        return False\n    else:\n        return True\n\n\ndef lib2to3_unparse(node: Node) -> str:\n    \"\"\"Given a lib2to3 node, return its string representation.\"\"\"\n    code = str(node)\n    return code\n\n\ndef parse_single_version(\n    src: str, version: Tuple[int, int], *, type_comments: bool\n) -> Union[ast.AST, ast3.AST]:\n    filename = \"<unknown>\"\n    # typed-ast is needed because of feature version limitations in the builtin ast 3.8>\n    if sys.version_info >= (3, 8) and version >= (3,):\n        return ast.parse(\n            src, filename, feature_version=version, type_comments=type_comments\n        )\n\n    if _IS_PYPY:\n        # PyPy 3.7 doesn't support type comment tracking which is not ideal, but there's\n        # not much we can do as typed-ast won't work either.\n        if sys.version_info >= (3, 8):\n            return ast3.parse(src, filename, type_comments=type_comments)\n        else:\n            return ast3.parse(src, filename)\n    else:\n        if type_comments:\n            # Typed-ast is guaranteed to be used here and automatically tracks type\n            # comments separately.\n            return ast3.parse(src, filename, feature_version=version[1])\n        else:\n            return ast.parse(src, filename)\n\n\ndef parse_ast(src: str) -> Union[ast.AST, ast3.AST]:\n    # TODO: support Python 4+ ;)\n    versions = [(3, minor) for minor in range(3, sys.version_info[1] + 1)]\n\n    first_error = \"\"\n    for version in sorted(versions, reverse=True):\n        try:\n            return parse_single_version(src, version, type_comments=True)\n        except SyntaxError as e:\n            if not first_error:\n                first_error = str(e)\n\n    # Try to parse without type comments\n    for version in sorted(versions, reverse=True):\n        try:\n            return parse_single_version(src, version, type_comments=False)\n        except SyntaxError:\n            pass\n\n    raise SyntaxError(first_error)\n\n\nast3_AST: Final[Type[ast3.AST]] = ast3.AST\n\n\ndef _normalize(lineend: str, value: str) -> str:\n    # To normalize, we strip any leading and trailing space from\n    # each line...\n    stripped: List[str] = [i.strip() for i in value.splitlines()]\n    normalized = lineend.join(stripped)\n    # ...and remove any blank lines at the beginning and end of\n    # the whole string\n    return normalized.strip()\n\n\ndef stringify_ast(node: Union[ast.AST, ast3.AST], depth: int = 0) -> Iterator[str]:\n    \"\"\"Simple visitor generating strings to compare ASTs by content.\"\"\"\n\n    node = fixup_ast_constants(node)\n\n    yield f\"{'  ' * depth}{node.__class__.__name__}(\"\n\n    type_ignore_classes: Tuple[Type[Any], ...]\n    for field in sorted(node._fields):  # noqa: F402\n        # TypeIgnore will not be present using pypy < 3.8, so need for this\n        if not (_IS_PYPY and sys.version_info < (3, 8)):\n            # TypeIgnore has only one field 'lineno' which breaks this comparison\n            type_ignore_classes = (ast3.TypeIgnore,)\n            if sys.version_info >= (3, 8):\n                type_ignore_classes += (ast.TypeIgnore,)\n            if isinstance(node, type_ignore_classes):\n                break\n\n        try:\n            value: object = getattr(node, field)\n        except AttributeError:\n            continue\n\n        yield f\"{'  ' * (depth+1)}{field}=\"\n\n        if isinstance(value, list):\n            for item in value:\n                # Ignore nested tuples within del statements, because we may insert\n                # parentheses and they change the AST.\n                if (\n                    field == \"targets\"\n                    and isinstance(node, (ast.Delete, ast3.Delete))\n                    and isinstance(item, (ast.Tuple, ast3.Tuple))\n                ):\n                    for elt in item.elts:\n                        yield from stringify_ast(elt, depth + 2)\n\n                elif isinstance(item, (ast.AST, ast3.AST)):\n                    yield from stringify_ast(item, depth + 2)\n\n        # Note that we are referencing the typed-ast ASTs via global variables and not\n        # direct module attribute accesses because that breaks mypyc. It's probably\n        # something to do with the ast3 variables being marked as Any leading\n        # mypy to think this branch is always taken, leaving the rest of the code\n        # unanalyzed. Tighting up the types for the typed-ast AST types avoids the\n        # mypyc crash.\n        elif isinstance(value, (ast.AST, ast3_AST)):\n            yield from stringify_ast(value, depth + 2)\n\n        else:\n            normalized: object\n            # Constant strings may be indented across newlines, if they are\n            # docstrings; fold spaces after newlines when comparing. Similarly,\n            # trailing and leading space may be removed.\n            if (\n                isinstance(node, ast.Constant)\n                and field == \"value\"\n                and isinstance(value, str)\n            ):\n                normalized = _normalize(\"\\n\", value)\n            else:\n                normalized = value\n            yield f\"{'  ' * (depth+2)}{normalized!r},  # {value.__class__.__name__}\"\n\n    yield f\"{'  ' * depth})  # /{node.__class__.__name__}\"\n\n\ndef fixup_ast_constants(node: Union[ast.AST, ast3.AST]) -> Union[ast.AST, ast3.AST]:\n    \"\"\"Map ast nodes deprecated in 3.8 to Constant.\"\"\"\n    if isinstance(node, (ast.Str, ast3.Str, ast.Bytes, ast3.Bytes)):\n        return ast.Constant(value=node.s)\n\n    if isinstance(node, (ast.Num, ast3.Num)):\n        return ast.Constant(value=node.n)\n\n    if isinstance(node, (ast.NameConstant, ast3.NameConstant)):\n        return ast.Constant(value=node.value)\n\n    return node\n", "path": "src/black/parsing.py"}]}