We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Stripping non-ASCII characters
**Describe the bug**
We have some Fortran files that contain non-ASCII characters in comments. They compile fine but when used with meson, I get errors in `depscan.py`:
```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 3329: invalid continuation byte
```

Instead of writing a custom script that strips out these non-ASCII characters, I wonder if there's a way to do it from within meson. Alternatively, is it possible to just bypass this check or make it skip comments?



**To Reproduce**
I'm working off of this pyOptSparse build at mdolab/pyoptsparse#300. The offending Fortran source file is unfortunately not publicly available, but any Fortran file with a non-ASCII character in a comment line should do.

**Expected behavior**
First, I think better debugging info would be helpful. It's not clear from the message which file was the issue, and I had to check with `chardetect` to figure it out.

I think the best thing would be for meson to allow these characters to exist, since the compiler has no problems with it. If that's not possible, then I was wondering if meson provides any utilities in stripping characters from files, as that seems to be fairly useful.

**system parameters**
* Is this a [cross build](https://mesonbuild.com/Cross-compilation.html) or just a plain native build (for the same computer)? Native build
* what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) Manjaro 21.2.6
* what Python version are you using? Python 3.9
* what `meson --version` 0.63.0
* what `ninja --version` if it's a Ninja build  `1.10.2.git.kitware.jobserver-1`


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `mesonbuild/scripts/depscan.py`
Content:
```
1 # Copyright 2020 The Meson development team
2 
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 from __future__ import annotations
15 
16 import json
17 import os
18 import pathlib
19 import pickle
20 import re
21 import sys
22 import typing as T
23 
24 from ..backend.ninjabackend import ninja_quote
25 from ..compilers.compilers import lang_suffixes
26 
27 if T.TYPE_CHECKING:
28     from ..backend.ninjabackend import TargetDependencyScannerInfo
29 
30 CPP_IMPORT_RE = re.compile(r'\w*import ([a-zA-Z0-9]+);')
31 CPP_EXPORT_RE = re.compile(r'\w*export module ([a-zA-Z0-9]+);')
32 
33 FORTRAN_INCLUDE_PAT = r"^\s*include\s*['\"](\w+\.\w+)['\"]"
34 FORTRAN_MODULE_PAT = r"^\s*\bmodule\b\s+(\w+)\s*(?:!+.*)*$"
35 FORTRAN_SUBMOD_PAT = r"^\s*\bsubmodule\b\s*\((\w+:?\w+)\)\s*(\w+)"
36 FORTRAN_USE_PAT = r"^\s*use,?\s*(?:non_intrinsic)?\s*(?:::)?\s*(\w+)"
37 
38 FORTRAN_MODULE_RE = re.compile(FORTRAN_MODULE_PAT, re.IGNORECASE)
39 FORTRAN_SUBMOD_RE = re.compile(FORTRAN_SUBMOD_PAT, re.IGNORECASE)
40 FORTRAN_USE_RE = re.compile(FORTRAN_USE_PAT, re.IGNORECASE)
41 
42 class DependencyScanner:
43     def __init__(self, pickle_file: str, outfile: str, sources: T.List[str]):
44         with open(pickle_file, 'rb') as pf:
45             self.target_data: TargetDependencyScannerInfo = pickle.load(pf)
46         self.outfile = outfile
47         self.sources = sources
48         self.provided_by: T.Dict[str, str] = {}
49         self.exports: T.Dict[str, str] = {}
50         self.needs: T.Dict[str, T.List[str]] = {}
51         self.sources_with_exports: T.List[str] = []
52 
53     def scan_file(self, fname: str) -> None:
54         suffix = os.path.splitext(fname)[1][1:].lower()
55         if suffix in lang_suffixes['fortran']:
56             self.scan_fortran_file(fname)
57         elif suffix in lang_suffixes['cpp']:
58             self.scan_cpp_file(fname)
59         else:
60             sys.exit(f'Can not scan files with suffix .{suffix}.')
61 
62     def scan_fortran_file(self, fname: str) -> None:
63         fpath = pathlib.Path(fname)
64         modules_in_this_file = set()
65         for line in fpath.read_text(encoding='utf-8').split('\n'):
66             import_match = FORTRAN_USE_RE.match(line)
67             export_match = FORTRAN_MODULE_RE.match(line)
68             submodule_export_match = FORTRAN_SUBMOD_RE.match(line)
69             if import_match:
70                 needed = import_match.group(1).lower()
71                 # In Fortran you have an using declaration also for the module
72                 # you define in the same file. Prevent circular dependencies.
73                 if needed not in modules_in_this_file:
74                     if fname in self.needs:
75                         self.needs[fname].append(needed)
76                     else:
77                         self.needs[fname] = [needed]
78             if export_match:
79                 exported_module = export_match.group(1).lower()
80                 assert exported_module not in modules_in_this_file
81                 modules_in_this_file.add(exported_module)
82                 if exported_module in self.provided_by:
83                     raise RuntimeError(f'Multiple files provide module {exported_module}.')
84                 self.sources_with_exports.append(fname)
85                 self.provided_by[exported_module] = fname
86                 self.exports[fname] = exported_module
87             if submodule_export_match:
88                 # Store submodule "Foo" "Bar" as "foo:bar".
89                 # A submodule declaration can be both an import and an export declaration:
90                 #
91                 # submodule (a1:a2) a3
92                 #  - requires a1@a2.smod
93                 #  - produces a1@a3.smod
94                 parent_module_name_full = submodule_export_match.group(1).lower()
95                 parent_module_name = parent_module_name_full.split(':')[0]
96                 submodule_name = submodule_export_match.group(2).lower()
97                 concat_name = f'{parent_module_name}:{submodule_name}'
98                 self.sources_with_exports.append(fname)
99                 self.provided_by[concat_name] = fname
100                 self.exports[fname] = concat_name
101                 # Fortran requires that the immediate parent module must be built
102                 # before the current one. Thus:
103                 #
104                 # submodule (parent) parent   <- requires parent.mod (really parent.smod, but they are created at the same time)
105                 # submodule (a1:a2) a3        <- requires a1@a2.smod
106                 #
107                 # a3 does not depend on the a1 parent module directly, only transitively.
108                 if fname in self.needs:
109                     self.needs[fname].append(parent_module_name_full)
110                 else:
111                     self.needs[fname] = [parent_module_name_full]
112 
113     def scan_cpp_file(self, fname: str) -> None:
114         fpath = pathlib.Path(fname)
115         for line in fpath.read_text(encoding='utf-8').split('\n'):
116             import_match = CPP_IMPORT_RE.match(line)
117             export_match = CPP_EXPORT_RE.match(line)
118             if import_match:
119                 needed = import_match.group(1)
120                 if fname in self.needs:
121                     self.needs[fname].append(needed)
122                 else:
123                     self.needs[fname] = [needed]
124             if export_match:
125                 exported_module = export_match.group(1)
126                 if exported_module in self.provided_by:
127                     raise RuntimeError(f'Multiple files provide module {exported_module}.')
128                 self.sources_with_exports.append(fname)
129                 self.provided_by[exported_module] = fname
130                 self.exports[fname] = exported_module
131 
132     def objname_for(self, src: str) -> str:
133         objname = self.target_data.source2object[src]
134         assert isinstance(objname, str)
135         return objname
136 
137     def module_name_for(self, src: str) -> str:
138         suffix = os.path.splitext(src)[1][1:].lower()
139         if suffix in lang_suffixes['fortran']:
140             exported = self.exports[src]
141             # Module foo:bar goes to a file name foo@bar.smod
142             # Module Foo goes to a file name foo.mod
143             namebase = exported.replace(':', '@')
144             if ':' in exported:
145                 extension = 'smod'
146             else:
147                 extension = 'mod'
148             return os.path.join(self.target_data.private_dir, f'{namebase}.{extension}')
149         elif suffix in lang_suffixes['cpp']:
150             return '{}.ifc'.format(self.exports[src])
151         else:
152             raise RuntimeError('Unreachable code.')
153 
154     def scan(self) -> int:
155         for s in self.sources:
156             self.scan_file(s)
157         with open(self.outfile, 'w', encoding='utf-8') as ofile:
158             ofile.write('ninja_dyndep_version = 1\n')
159             for src in self.sources:
160                 objfilename = self.objname_for(src)
161                 mods_and_submods_needed = []
162                 module_files_generated = []
163                 module_files_needed = []
164                 if src in self.sources_with_exports:
165                     module_files_generated.append(self.module_name_for(src))
166                 if src in self.needs:
167                     for modname in self.needs[src]:
168                         if modname not in self.provided_by:
169                             # Nothing provides this module, we assume that it
170                             # comes from a dependency library somewhere and is
171                             # already built by the time this compilation starts.
172                             pass
173                         else:
174                             mods_and_submods_needed.append(modname)
175 
176                 for modname in mods_and_submods_needed:
177                     provider_src = self.provided_by[modname]
178                     provider_modfile = self.module_name_for(provider_src)
179                     # Prune self-dependencies
180                     if provider_src != src:
181                         module_files_needed.append(provider_modfile)
182 
183                 quoted_objfilename = ninja_quote(objfilename, True)
184                 quoted_module_files_generated = [ninja_quote(x, True) for x in module_files_generated]
185                 quoted_module_files_needed = [ninja_quote(x, True) for x in module_files_needed]
186                 if quoted_module_files_generated:
187                     mod_gen = '| ' + ' '.join(quoted_module_files_generated)
188                 else:
189                     mod_gen = ''
190                 if quoted_module_files_needed:
191                     mod_dep = '| ' + ' '.join(quoted_module_files_needed)
192                 else:
193                     mod_dep = ''
194                 build_line = 'build {} {}: dyndep {}'.format(quoted_objfilename,
195                                                              mod_gen,
196                                                              mod_dep)
197                 ofile.write(build_line + '\n')
198         return 0
199 
200 def run(args: T.List[str]) -> int:
201     assert len(args) == 3, 'got wrong number of arguments!'
202     pickle_file, outfile, jsonfile = args
203     with open(jsonfile, encoding='utf-8') as f:
204         sources = json.load(f)
205     scanner = DependencyScanner(pickle_file, outfile, sources)
206     return scanner.scan()
207
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/mesonbuild/scripts/depscan.py b/mesonbuild/scripts/depscan.py
--- a/mesonbuild/scripts/depscan.py
+++ b/mesonbuild/scripts/depscan.py
@@ -62,7 +62,7 @@
     def scan_fortran_file(self, fname: str) -> None:
         fpath = pathlib.Path(fname)
         modules_in_this_file = set()
-        for line in fpath.read_text(encoding='utf-8').split('\n'):
+        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\n'):
             import_match = FORTRAN_USE_RE.match(line)
             export_match = FORTRAN_MODULE_RE.match(line)
             submodule_export_match = FORTRAN_SUBMOD_RE.match(line)
@@ -112,7 +112,7 @@
 
     def scan_cpp_file(self, fname: str) -> None:
         fpath = pathlib.Path(fname)
-        for line in fpath.read_text(encoding='utf-8').split('\n'):
+        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\n'):
             import_match = CPP_IMPORT_RE.match(line)
             export_match = CPP_EXPORT_RE.match(line)
             if import_match:

{"golden_diff": "diff --git a/mesonbuild/scripts/depscan.py b/mesonbuild/scripts/depscan.py\n--- a/mesonbuild/scripts/depscan.py\n+++ b/mesonbuild/scripts/depscan.py\n@@ -62,7 +62,7 @@\n     def scan_fortran_file(self, fname: str) -> None:\n         fpath = pathlib.Path(fname)\n         modules_in_this_file = set()\n-        for line in fpath.read_text(encoding='utf-8').split('\\n'):\n+        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\\n'):\n             import_match = FORTRAN_USE_RE.match(line)\n             export_match = FORTRAN_MODULE_RE.match(line)\n             submodule_export_match = FORTRAN_SUBMOD_RE.match(line)\n@@ -112,7 +112,7 @@\n \n     def scan_cpp_file(self, fname: str) -> None:\n         fpath = pathlib.Path(fname)\n-        for line in fpath.read_text(encoding='utf-8').split('\\n'):\n+        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\\n'):\n             import_match = CPP_IMPORT_RE.match(line)\n             export_match = CPP_EXPORT_RE.match(line)\n             if import_match:\n", "issue": "Stripping non-ASCII characters\n**Describe the bug**\r\nWe have some Fortran files that contain non-ASCII characters in comments. They compile fine but when used with meson, I get errors in `depscan.py`:\r\n```\r\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 3329: invalid continuation byte\r\n```\r\n\r\nInstead of writing a custom script that strips out these non-ASCII characters, I wonder if there's a way to do it from within meson. Alternatively, is it possible to just bypass this check or make it skip comments?\r\n\r\n\r\n\r\n**To Reproduce**\r\nI'm working off of this pyOptSparse build at mdolab/pyoptsparse#300. The offending Fortran source file is unfortunately not publicly available, but any Fortran file with a non-ASCII character in a comment line should do.\r\n\r\n**Expected behavior**\r\nFirst, I think better debugging info would be helpful. It's not clear from the message which file was the issue, and I had to check with `chardetect` to figure it out.\r\n\r\nI think the best thing would be for meson to allow these characters to exist, since the compiler has no problems with it. If that's not possible, then I was wondering if meson provides any utilities in stripping characters from files, as that seems to be fairly useful.\r\n\r\n**system parameters**\r\n* Is this a [cross build](https://mesonbuild.com/Cross-compilation.html) or just a plain native build (for the same computer)? Native build\r\n* what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.) Manjaro 21.2.6\r\n* what Python version are you using? Python 3.9\r\n* what `meson --version` 0.63.0\r\n* what `ninja --version` if it's a Ninja build  `1.10.2.git.kitware.jobserver-1`\r\n\n", "before_files": [{"content": "# Copyright 2020 The Meson development team\n\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n\n#     http://www.apache.org/licenses/LICENSE-2.0\n\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import annotations\n\nimport json\nimport os\nimport pathlib\nimport pickle\nimport re\nimport sys\nimport typing as T\n\nfrom ..backend.ninjabackend import ninja_quote\nfrom ..compilers.compilers import lang_suffixes\n\nif T.TYPE_CHECKING:\n    from ..backend.ninjabackend import TargetDependencyScannerInfo\n\nCPP_IMPORT_RE = re.compile(r'\\w*import ([a-zA-Z0-9]+);')\nCPP_EXPORT_RE = re.compile(r'\\w*export module ([a-zA-Z0-9]+);')\n\nFORTRAN_INCLUDE_PAT = r\"^\\s*include\\s*['\\\"](\\w+\\.\\w+)['\\\"]\"\nFORTRAN_MODULE_PAT = r\"^\\s*\\bmodule\\b\\s+(\\w+)\\s*(?:!+.*)*$\"\nFORTRAN_SUBMOD_PAT = r\"^\\s*\\bsubmodule\\b\\s*\\((\\w+:?\\w+)\\)\\s*(\\w+)\"\nFORTRAN_USE_PAT = r\"^\\s*use,?\\s*(?:non_intrinsic)?\\s*(?:::)?\\s*(\\w+)\"\n\nFORTRAN_MODULE_RE = re.compile(FORTRAN_MODULE_PAT, re.IGNORECASE)\nFORTRAN_SUBMOD_RE = re.compile(FORTRAN_SUBMOD_PAT, re.IGNORECASE)\nFORTRAN_USE_RE = re.compile(FORTRAN_USE_PAT, re.IGNORECASE)\n\nclass DependencyScanner:\n    def __init__(self, pickle_file: str, outfile: str, sources: T.List[str]):\n        with open(pickle_file, 'rb') as pf:\n            self.target_data: TargetDependencyScannerInfo = pickle.load(pf)\n        self.outfile = outfile\n        self.sources = sources\n        self.provided_by: T.Dict[str, str] = {}\n        self.exports: T.Dict[str, str] = {}\n        self.needs: T.Dict[str, T.List[str]] = {}\n        self.sources_with_exports: T.List[str] = []\n\n    def scan_file(self, fname: str) -> None:\n        suffix = os.path.splitext(fname)[1][1:].lower()\n        if suffix in lang_suffixes['fortran']:\n            self.scan_fortran_file(fname)\n        elif suffix in lang_suffixes['cpp']:\n            self.scan_cpp_file(fname)\n        else:\n            sys.exit(f'Can not scan files with suffix .{suffix}.')\n\n    def scan_fortran_file(self, fname: str) -> None:\n        fpath = pathlib.Path(fname)\n        modules_in_this_file = set()\n        for line in fpath.read_text(encoding='utf-8').split('\\n'):\n            import_match = FORTRAN_USE_RE.match(line)\n            export_match = FORTRAN_MODULE_RE.match(line)\n            submodule_export_match = FORTRAN_SUBMOD_RE.match(line)\n            if import_match:\n                needed = import_match.group(1).lower()\n                # In Fortran you have an using declaration also for the module\n                # you define in the same file. Prevent circular dependencies.\n                if needed not in modules_in_this_file:\n                    if fname in self.needs:\n                        self.needs[fname].append(needed)\n                    else:\n                        self.needs[fname] = [needed]\n            if export_match:\n                exported_module = export_match.group(1).lower()\n                assert exported_module not in modules_in_this_file\n                modules_in_this_file.add(exported_module)\n                if exported_module in self.provided_by:\n                    raise RuntimeError(f'Multiple files provide module {exported_module}.')\n                self.sources_with_exports.append(fname)\n                self.provided_by[exported_module] = fname\n                self.exports[fname] = exported_module\n            if submodule_export_match:\n                # Store submodule \"Foo\" \"Bar\" as \"foo:bar\".\n                # A submodule declaration can be both an import and an export declaration:\n                #\n                # submodule (a1:a2) a3\n                #  - requires a1@a2.smod\n                #  - produces a1@a3.smod\n                parent_module_name_full = submodule_export_match.group(1).lower()\n                parent_module_name = parent_module_name_full.split(':')[0]\n                submodule_name = submodule_export_match.group(2).lower()\n                concat_name = f'{parent_module_name}:{submodule_name}'\n                self.sources_with_exports.append(fname)\n                self.provided_by[concat_name] = fname\n                self.exports[fname] = concat_name\n                # Fortran requires that the immediate parent module must be built\n                # before the current one. Thus:\n                #\n                # submodule (parent) parent   <- requires parent.mod (really parent.smod, but they are created at the same time)\n                # submodule (a1:a2) a3        <- requires a1@a2.smod\n                #\n                # a3 does not depend on the a1 parent module directly, only transitively.\n                if fname in self.needs:\n                    self.needs[fname].append(parent_module_name_full)\n                else:\n                    self.needs[fname] = [parent_module_name_full]\n\n    def scan_cpp_file(self, fname: str) -> None:\n        fpath = pathlib.Path(fname)\n        for line in fpath.read_text(encoding='utf-8').split('\\n'):\n            import_match = CPP_IMPORT_RE.match(line)\n            export_match = CPP_EXPORT_RE.match(line)\n            if import_match:\n                needed = import_match.group(1)\n                if fname in self.needs:\n                    self.needs[fname].append(needed)\n                else:\n                    self.needs[fname] = [needed]\n            if export_match:\n                exported_module = export_match.group(1)\n                if exported_module in self.provided_by:\n                    raise RuntimeError(f'Multiple files provide module {exported_module}.')\n                self.sources_with_exports.append(fname)\n                self.provided_by[exported_module] = fname\n                self.exports[fname] = exported_module\n\n    def objname_for(self, src: str) -> str:\n        objname = self.target_data.source2object[src]\n        assert isinstance(objname, str)\n        return objname\n\n    def module_name_for(self, src: str) -> str:\n        suffix = os.path.splitext(src)[1][1:].lower()\n        if suffix in lang_suffixes['fortran']:\n            exported = self.exports[src]\n            # Module foo:bar goes to a file name foo@bar.smod\n            # Module Foo goes to a file name foo.mod\n            namebase = exported.replace(':', '@')\n            if ':' in exported:\n                extension = 'smod'\n            else:\n                extension = 'mod'\n            return os.path.join(self.target_data.private_dir, f'{namebase}.{extension}')\n        elif suffix in lang_suffixes['cpp']:\n            return '{}.ifc'.format(self.exports[src])\n        else:\n            raise RuntimeError('Unreachable code.')\n\n    def scan(self) -> int:\n        for s in self.sources:\n            self.scan_file(s)\n        with open(self.outfile, 'w', encoding='utf-8') as ofile:\n            ofile.write('ninja_dyndep_version = 1\\n')\n            for src in self.sources:\n                objfilename = self.objname_for(src)\n                mods_and_submods_needed = []\n                module_files_generated = []\n                module_files_needed = []\n                if src in self.sources_with_exports:\n                    module_files_generated.append(self.module_name_for(src))\n                if src in self.needs:\n                    for modname in self.needs[src]:\n                        if modname not in self.provided_by:\n                            # Nothing provides this module, we assume that it\n                            # comes from a dependency library somewhere and is\n                            # already built by the time this compilation starts.\n                            pass\n                        else:\n                            mods_and_submods_needed.append(modname)\n\n                for modname in mods_and_submods_needed:\n                    provider_src = self.provided_by[modname]\n                    provider_modfile = self.module_name_for(provider_src)\n                    # Prune self-dependencies\n                    if provider_src != src:\n                        module_files_needed.append(provider_modfile)\n\n                quoted_objfilename = ninja_quote(objfilename, True)\n                quoted_module_files_generated = [ninja_quote(x, True) for x in module_files_generated]\n                quoted_module_files_needed = [ninja_quote(x, True) for x in module_files_needed]\n                if quoted_module_files_generated:\n                    mod_gen = '| ' + ' '.join(quoted_module_files_generated)\n                else:\n                    mod_gen = ''\n                if quoted_module_files_needed:\n                    mod_dep = '| ' + ' '.join(quoted_module_files_needed)\n                else:\n                    mod_dep = ''\n                build_line = 'build {} {}: dyndep {}'.format(quoted_objfilename,\n                                                             mod_gen,\n                                                             mod_dep)\n                ofile.write(build_line + '\\n')\n        return 0\n\ndef run(args: T.List[str]) -> int:\n    assert len(args) == 3, 'got wrong number of arguments!'\n    pickle_file, outfile, jsonfile = args\n    with open(jsonfile, encoding='utf-8') as f:\n        sources = json.load(f)\n    scanner = DependencyScanner(pickle_file, outfile, sources)\n    return scanner.scan()\n", "path": "mesonbuild/scripts/depscan.py"}], "after_files": [{"content": "# Copyright 2020 The Meson development team\n\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n\n#     http://www.apache.org/licenses/LICENSE-2.0\n\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nfrom __future__ import annotations\n\nimport json\nimport os\nimport pathlib\nimport pickle\nimport re\nimport sys\nimport typing as T\n\nfrom ..backend.ninjabackend import ninja_quote\nfrom ..compilers.compilers import lang_suffixes\n\nif T.TYPE_CHECKING:\n    from ..backend.ninjabackend import TargetDependencyScannerInfo\n\nCPP_IMPORT_RE = re.compile(r'\\w*import ([a-zA-Z0-9]+);')\nCPP_EXPORT_RE = re.compile(r'\\w*export module ([a-zA-Z0-9]+);')\n\nFORTRAN_INCLUDE_PAT = r\"^\\s*include\\s*['\\\"](\\w+\\.\\w+)['\\\"]\"\nFORTRAN_MODULE_PAT = r\"^\\s*\\bmodule\\b\\s+(\\w+)\\s*(?:!+.*)*$\"\nFORTRAN_SUBMOD_PAT = r\"^\\s*\\bsubmodule\\b\\s*\\((\\w+:?\\w+)\\)\\s*(\\w+)\"\nFORTRAN_USE_PAT = r\"^\\s*use,?\\s*(?:non_intrinsic)?\\s*(?:::)?\\s*(\\w+)\"\n\nFORTRAN_MODULE_RE = re.compile(FORTRAN_MODULE_PAT, re.IGNORECASE)\nFORTRAN_SUBMOD_RE = re.compile(FORTRAN_SUBMOD_PAT, re.IGNORECASE)\nFORTRAN_USE_RE = re.compile(FORTRAN_USE_PAT, re.IGNORECASE)\n\nclass DependencyScanner:\n    def __init__(self, pickle_file: str, outfile: str, sources: T.List[str]):\n        with open(pickle_file, 'rb') as pf:\n            self.target_data: TargetDependencyScannerInfo = pickle.load(pf)\n        self.outfile = outfile\n        self.sources = sources\n        self.provided_by: T.Dict[str, str] = {}\n        self.exports: T.Dict[str, str] = {}\n        self.needs: T.Dict[str, T.List[str]] = {}\n        self.sources_with_exports: T.List[str] = []\n\n    def scan_file(self, fname: str) -> None:\n        suffix = os.path.splitext(fname)[1][1:].lower()\n        if suffix in lang_suffixes['fortran']:\n            self.scan_fortran_file(fname)\n        elif suffix in lang_suffixes['cpp']:\n            self.scan_cpp_file(fname)\n        else:\n            sys.exit(f'Can not scan files with suffix .{suffix}.')\n\n    def scan_fortran_file(self, fname: str) -> None:\n        fpath = pathlib.Path(fname)\n        modules_in_this_file = set()\n        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\\n'):\n            import_match = FORTRAN_USE_RE.match(line)\n            export_match = FORTRAN_MODULE_RE.match(line)\n            submodule_export_match = FORTRAN_SUBMOD_RE.match(line)\n            if import_match:\n                needed = import_match.group(1).lower()\n                # In Fortran you have an using declaration also for the module\n                # you define in the same file. Prevent circular dependencies.\n                if needed not in modules_in_this_file:\n                    if fname in self.needs:\n                        self.needs[fname].append(needed)\n                    else:\n                        self.needs[fname] = [needed]\n            if export_match:\n                exported_module = export_match.group(1).lower()\n                assert exported_module not in modules_in_this_file\n                modules_in_this_file.add(exported_module)\n                if exported_module in self.provided_by:\n                    raise RuntimeError(f'Multiple files provide module {exported_module}.')\n                self.sources_with_exports.append(fname)\n                self.provided_by[exported_module] = fname\n                self.exports[fname] = exported_module\n            if submodule_export_match:\n                # Store submodule \"Foo\" \"Bar\" as \"foo:bar\".\n                # A submodule declaration can be both an import and an export declaration:\n                #\n                # submodule (a1:a2) a3\n                #  - requires a1@a2.smod\n                #  - produces a1@a3.smod\n                parent_module_name_full = submodule_export_match.group(1).lower()\n                parent_module_name = parent_module_name_full.split(':')[0]\n                submodule_name = submodule_export_match.group(2).lower()\n                concat_name = f'{parent_module_name}:{submodule_name}'\n                self.sources_with_exports.append(fname)\n                self.provided_by[concat_name] = fname\n                self.exports[fname] = concat_name\n                # Fortran requires that the immediate parent module must be built\n                # before the current one. Thus:\n                #\n                # submodule (parent) parent   <- requires parent.mod (really parent.smod, but they are created at the same time)\n                # submodule (a1:a2) a3        <- requires a1@a2.smod\n                #\n                # a3 does not depend on the a1 parent module directly, only transitively.\n                if fname in self.needs:\n                    self.needs[fname].append(parent_module_name_full)\n                else:\n                    self.needs[fname] = [parent_module_name_full]\n\n    def scan_cpp_file(self, fname: str) -> None:\n        fpath = pathlib.Path(fname)\n        for line in fpath.read_text(encoding='utf-8', errors='ignore').split('\\n'):\n            import_match = CPP_IMPORT_RE.match(line)\n            export_match = CPP_EXPORT_RE.match(line)\n            if import_match:\n                needed = import_match.group(1)\n                if fname in self.needs:\n                    self.needs[fname].append(needed)\n                else:\n                    self.needs[fname] = [needed]\n            if export_match:\n                exported_module = export_match.group(1)\n                if exported_module in self.provided_by:\n                    raise RuntimeError(f'Multiple files provide module {exported_module}.')\n                self.sources_with_exports.append(fname)\n                self.provided_by[exported_module] = fname\n                self.exports[fname] = exported_module\n\n    def objname_for(self, src: str) -> str:\n        objname = self.target_data.source2object[src]\n        assert isinstance(objname, str)\n        return objname\n\n    def module_name_for(self, src: str) -> str:\n        suffix = os.path.splitext(src)[1][1:].lower()\n        if suffix in lang_suffixes['fortran']:\n            exported = self.exports[src]\n            # Module foo:bar goes to a file name foo@bar.smod\n            # Module Foo goes to a file name foo.mod\n            namebase = exported.replace(':', '@')\n            if ':' in exported:\n                extension = 'smod'\n            else:\n                extension = 'mod'\n            return os.path.join(self.target_data.private_dir, f'{namebase}.{extension}')\n        elif suffix in lang_suffixes['cpp']:\n            return '{}.ifc'.format(self.exports[src])\n        else:\n            raise RuntimeError('Unreachable code.')\n\n    def scan(self) -> int:\n        for s in self.sources:\n            self.scan_file(s)\n        with open(self.outfile, 'w', encoding='utf-8') as ofile:\n            ofile.write('ninja_dyndep_version = 1\\n')\n            for src in self.sources:\n                objfilename = self.objname_for(src)\n                mods_and_submods_needed = []\n                module_files_generated = []\n                module_files_needed = []\n                if src in self.sources_with_exports:\n                    module_files_generated.append(self.module_name_for(src))\n                if src in self.needs:\n                    for modname in self.needs[src]:\n                        if modname not in self.provided_by:\n                            # Nothing provides this module, we assume that it\n                            # comes from a dependency library somewhere and is\n                            # already built by the time this compilation starts.\n                            pass\n                        else:\n                            mods_and_submods_needed.append(modname)\n\n                for modname in mods_and_submods_needed:\n                    provider_src = self.provided_by[modname]\n                    provider_modfile = self.module_name_for(provider_src)\n                    # Prune self-dependencies\n                    if provider_src != src:\n                        module_files_needed.append(provider_modfile)\n\n                quoted_objfilename = ninja_quote(objfilename, True)\n                quoted_module_files_generated = [ninja_quote(x, True) for x in module_files_generated]\n                quoted_module_files_needed = [ninja_quote(x, True) for x in module_files_needed]\n                if quoted_module_files_generated:\n                    mod_gen = '| ' + ' '.join(quoted_module_files_generated)\n                else:\n                    mod_gen = ''\n                if quoted_module_files_needed:\n                    mod_dep = '| ' + ' '.join(quoted_module_files_needed)\n                else:\n                    mod_dep = ''\n                build_line = 'build {} {}: dyndep {}'.format(quoted_objfilename,\n                                                             mod_gen,\n                                                             mod_dep)\n                ofile.write(build_line + '\\n')\n        return 0\n\ndef run(args: T.List[str]) -> int:\n    assert len(args) == 3, 'got wrong number of arguments!'\n    pickle_file, outfile, jsonfile = args\n    with open(jsonfile, encoding='utf-8') as f:\n        sources = json.load(f)\n    scanner = DependencyScanner(pickle_file, outfile, sources)\n    return scanner.scan()\n", "path": "mesonbuild/scripts/depscan.py"}]}