We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Logging output is much more verbose when running with n_jobs > 1 vs n_jobs=1
After #799, I still see lots of output when Reports are created – but only if `n_jobs > 1`. For `n_jobs = 1`, things are working as expected.

```
❯ mne_bids_pipeline --config=./bids_pipeline_config.py --n_jobs=1
┌────────┬ Welcome aboard MNE-BIDS-Pipeline! 👋 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:26│ 📝 Using configuration: ./bids_pipeline_config.py
│16:53:26│ ❌ Overriding config.n_jobs = 1
└────────┴ 
┌────────┬ init/_01_init_derivatives_dir ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:26│ ⏳️ Initializing output directories.
└────────┴ done (1s)
┌────────┬ init/_02_find_empty_room ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:26│ ⏩ Skipping, empty-room data only relevant for MEG …
└────────┴ done (1s)
┌────────┬ preprocessing/_01_data_quality ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:26│ ⏳️ sub-005 run-01 Initializing report HDF5 file
│16:53:26│ ⏳️ sub-005 run-01 Adding original raw data to report
│16:53:28│ ⏳️ sub-005 run-01 Adding config and sys info to report
```

```
❯ mne_bids_pipeline --config=./bids_pipeline_config.py --n_jobs=8
┌────────┬ Welcome aboard MNE-BIDS-Pipeline! 👋 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:50│ 📝 Using configuration: ./bids_pipeline_config.py
│16:53:50│ ❌ Overriding config.n_jobs = 8
└────────┴ 
┌────────┬ init/_01_init_derivatives_dir ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:50│ ✅ Output directories already exist …
└────────┴ done (1s)
┌────────┬ init/_02_find_empty_room ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│16:53:50│ ⏩ Skipping, empty-room data only relevant for MEG …
└────────┴ done (1s)
┌────────┬ preprocessing/_01_data_quality ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Embedding : jquery-3.6.0.min.js
Embedding : bootstrap.bundle.min.js
Embedding : bootstrap.min.css
Embedding : bootstrap-table/bootstrap-table.min.js
Embedding : bootstrap-table/bootstrap-table.min.css
Embedding : bootstrap-table/bootstrap-table-copy-rows.min.js
Embedding : bootstrap-table/bootstrap-table-export.min.js
Embedding : bootstrap-table/tableExport.min.js
Embedding : bootstrap-icons/bootstrap-icons.mne.min.css
Embedding : highlightjs/highlight.min.js
Embedding : highlightjs/atom-one-dark-reasonable.min.css
│16:53:51│ ⏳️ sub-005 run-01 Adding original raw data to report
│16:53:53│ ⏳️ sub-005 run-01 Adding config and sys info to report
│16:53:54│ ⏳️ sub-005 run-01 Saving report: /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-005/eeg/sub-005_task-tasin_report.html
Overwriting existing file.
Saving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-005/eeg/sub-005_task-tasin_report.h5
Saving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-005/eeg/sub-005_task-tasin_report.html
│16:53:54│ ⏳️ sub-102 run-01 Initializing report HDF5 file
Saving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.h5
Embedding : jquery-3.6.0.min.js
Embedding : bootstrap.bundle.min.js
Embedding : bootstrap.min.css
Embedding : bootstrap-table/bootstrap-table.min.js
Embedding : bootstrap-table/bootstrap-table.min.css
Embedding : bootstrap-table/bootstrap-table-copy-rows.min.js
Embedding : bootstrap-table/bootstrap-table-export.min.js
Embedding : bootstrap-table/tableExport.min.js
Embedding : bootstrap-icons/bootstrap-icons.mne.min.css
Embedding : highlightjs/highlight.min.js
Embedding : highlightjs/atom-one-dark-reasonable.min.css
Embedding : jquery-3.6.0.min.js
Embedding : bootstrap.bundle.min.js
Embedding : bootstrap.min.css
Embedding : bootstrap-table/bootstrap-table.min.js
Embedding : bootstrap-table/bootstrap-table.min.css
Embedding : bootstrap-table/bootstrap-table-copy-rows.min.js
Embedding : bootstrap-table/bootstrap-table-export.min.js
Embedding : bootstrap-table/tableExport.min.js
Embedding : bootstrap-icons/bootstrap-icons.mne.min.css
Embedding : highlightjs/highlight.min.js
Embedding : highlightjs/atom-one-dark-reasonable.min.css
│16:53:54│ ⏳️ sub-102 run-01 Adding original raw data to report
│16:53:54│ ⏳️ sub-005 run-04 Adding original raw data to report
│16:53:54│ ⏳️ sub-005 run-04 Adding config and sys info to report
│16:53:54│ ⏳️ sub-102 run-01 Adding config and sys info to report
│16:53:54│ ⏳️ sub-102 run-01 Saving report: /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.html
Overwriting existing file.
Saving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.h5
Saving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.html
Embedding : jquery-3.6.0.min.js
Embedding : bootstrap.bundle.min.js
Embedding : bootstrap.min.css
Embedding : bootstrap-table/bootstrap-table.min.js
Embedding : bootstrap-table/bootstrap-table.min.css
Embedding : bootstrap-table/bootstrap-table-copy-rows.min.js
Embedding : bootstrap-table/bootstrap-table-export.min.js
Embedding : bootstrap-table/tableExport.min.js
Embedding : bootstrap-icons/bootstrap-icons.mne.min.css
Embedding : highlightjs/highlight.min.js
Embedding : highlightjs/atom-one-dark-reasonable.min.css
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `mne_bids_pipeline/_parallel.py`
Content:
```
1 """Parallelization."""
2 
3 from typing import Literal, Callable
4 from types import SimpleNamespace
5 
6 import joblib
7 
8 from ._logging import logger, gen_log_kwargs, _is_testing
9 
10 
11 def get_n_jobs(*, exec_params: SimpleNamespace, log_override: bool = False) -> int:
12     n_jobs = exec_params.n_jobs
13     if n_jobs < 0:
14         n_cores = joblib.cpu_count()
15         n_jobs = min(n_cores + n_jobs + 1, n_cores)
16 
17     # Shim to allow overriding n_jobs for specific steps
18     if _is_testing() and hasattr(exec_params, "_n_jobs"):
19         from ._run import _get_step_path, _short_step_path
20 
21         step_path = _short_step_path(_get_step_path())
22         orig_n_jobs = n_jobs
23         n_jobs = exec_params._n_jobs.get(step_path, n_jobs)
24         if log_override and n_jobs != orig_n_jobs:
25             msg = f"Overriding n_jobs: {orig_n_jobs}→{n_jobs}"
26             logger.info(**gen_log_kwargs(message=msg, emoji="override"))
27     return n_jobs
28 
29 
30 dask_client = None
31 
32 
33 def setup_dask_client(*, exec_params: SimpleNamespace) -> None:
34     global dask_client
35 
36     import dask
37     from dask.distributed import Client
38 
39     if dask_client is not None:
40         return
41 
42     n_workers = get_n_jobs(exec_params=exec_params)
43     msg = f"Dask initializing with {n_workers} workers …"
44     logger.info(**gen_log_kwargs(message=msg, emoji="👾"))
45 
46     if exec_params.dask_temp_dir is None:
47         this_dask_temp_dir = exec_params.deriv_root / ".dask-worker-space"
48     else:
49         this_dask_temp_dir = exec_params.dask_temp_dir
50 
51     msg = f"Dask temporary directory: {this_dask_temp_dir}"
52     logger.info(**gen_log_kwargs(message=msg, emoji="📂"))
53     dask.config.set(
54         {
55             "temporary-directory": this_dask_temp_dir,
56             "distributed.worker.memory.pause": 0.8,
57             # fraction of memory that can be utilized before the nanny
58             # process will terminate the worker
59             "distributed.worker.memory.terminate": 1.0,
60             # TODO spilling to disk currently doesn't work reliably for us,
61             # as Dask cannot spill "unmanaged" memory – and most of what we
62             # see currently is, in fact, "unmanaged". Needs thorough
63             # investigation.
64             "distributed.worker.memory.spill": False,
65         }
66     )
67     client = Client(  # noqa: F841
68         memory_limit=exec_params.dask_worker_memory_limit,
69         n_workers=n_workers,
70         threads_per_worker=1,
71         name="mne-bids-pipeline",
72     )
73     client.auto_restart = False  # don't restart killed workers
74 
75     dashboard_url = client.dashboard_link
76     msg = "Dask client dashboard: " f"[link={dashboard_url}]{dashboard_url}[/link]"
77     logger.info(**gen_log_kwargs(message=msg, emoji="🌎"))
78 
79     if exec_params.dask_open_dashboard:
80         import webbrowser
81 
82         webbrowser.open(url=dashboard_url, autoraise=True)
83 
84     # Update global variable
85     dask_client = client
86 
87 
88 def get_parallel_backend_name(
89     *,
90     exec_params: SimpleNamespace,
91 ) -> Literal["dask", "loky"]:
92     if (
93         exec_params.parallel_backend == "loky"
94         or get_n_jobs(exec_params=exec_params) == 1
95     ):
96         backend = "loky"
97     elif exec_params.parallel_backend == "dask":
98         # Disable interactive plotting backend
99         import matplotlib
100 
101         matplotlib.use("Agg")
102         backend = "dask"
103     else:
104         # TODO: Move to value validation step
105         raise ValueError(f"Unknown parallel backend: {exec_params.parallel_backend}")
106 
107     return backend
108 
109 
110 def get_parallel_backend(exec_params: SimpleNamespace) -> joblib.parallel_backend:
111     import joblib
112 
113     backend = get_parallel_backend_name(exec_params=exec_params)
114     kwargs = {
115         "n_jobs": get_n_jobs(
116             exec_params=exec_params,
117             log_override=True,
118         )
119     }
120 
121     if backend == "loky":
122         kwargs["inner_max_num_threads"] = 1
123     else:
124         setup_dask_client(exec_params=exec_params)
125 
126     return joblib.parallel_backend(backend, **kwargs)
127 
128 
129 def parallel_func(func: Callable, *, exec_params: SimpleNamespace):
130     if get_parallel_backend_name(exec_params=exec_params) == "loky":
131         if get_n_jobs(exec_params=exec_params) == 1:
132             my_func = func
133             parallel = list
134         else:
135             from joblib import Parallel, delayed
136 
137             parallel = Parallel()
138             my_func = delayed(func)
139     else:  # Dask
140         from joblib import Parallel, delayed
141 
142         parallel = Parallel()
143         my_func = delayed(func)
144 
145     return parallel, my_func
146
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/mne_bids_pipeline/_parallel.py b/mne_bids_pipeline/_parallel.py
--- a/mne_bids_pipeline/_parallel.py
+++ b/mne_bids_pipeline/_parallel.py
@@ -4,6 +4,7 @@
 from types import SimpleNamespace
 
 import joblib
+from mne.utils import use_log_level, logger as mne_logger
 
 from ._logging import logger, gen_log_kwargs, _is_testing
 
@@ -127,19 +128,21 @@
 
 
 def parallel_func(func: Callable, *, exec_params: SimpleNamespace):
-    if get_parallel_backend_name(exec_params=exec_params) == "loky":
-        if get_n_jobs(exec_params=exec_params) == 1:
-            my_func = func
-            parallel = list
-        else:
-            from joblib import Parallel, delayed
-
-            parallel = Parallel()
-            my_func = delayed(func)
-    else:  # Dask
+    if (
+        get_parallel_backend_name(exec_params=exec_params) == "loky"
+        and get_n_jobs(exec_params=exec_params) == 1
+    ):
+        my_func = func
+        parallel = list
+    else:  # Dask or n_jobs > 1
         from joblib import Parallel, delayed
 
         parallel = Parallel()
-        my_func = delayed(func)
+
+        def run_verbose(*args, verbose=mne_logger.level, **kwargs):
+            with use_log_level(verbose=verbose):
+                return func(*args, **kwargs)
+
+        my_func = delayed(run_verbose)
 
     return parallel, my_func

{"golden_diff": "diff --git a/mne_bids_pipeline/_parallel.py b/mne_bids_pipeline/_parallel.py\n--- a/mne_bids_pipeline/_parallel.py\n+++ b/mne_bids_pipeline/_parallel.py\n@@ -4,6 +4,7 @@\n from types import SimpleNamespace\n \n import joblib\n+from mne.utils import use_log_level, logger as mne_logger\n \n from ._logging import logger, gen_log_kwargs, _is_testing\n \n@@ -127,19 +128,21 @@\n \n \n def parallel_func(func: Callable, *, exec_params: SimpleNamespace):\n-    if get_parallel_backend_name(exec_params=exec_params) == \"loky\":\n-        if get_n_jobs(exec_params=exec_params) == 1:\n-            my_func = func\n-            parallel = list\n-        else:\n-            from joblib import Parallel, delayed\n-\n-            parallel = Parallel()\n-            my_func = delayed(func)\n-    else:  # Dask\n+    if (\n+        get_parallel_backend_name(exec_params=exec_params) == \"loky\"\n+        and get_n_jobs(exec_params=exec_params) == 1\n+    ):\n+        my_func = func\n+        parallel = list\n+    else:  # Dask or n_jobs > 1\n         from joblib import Parallel, delayed\n \n         parallel = Parallel()\n-        my_func = delayed(func)\n+\n+        def run_verbose(*args, verbose=mne_logger.level, **kwargs):\n+            with use_log_level(verbose=verbose):\n+                return func(*args, **kwargs)\n+\n+        my_func = delayed(run_verbose)\n \n     return parallel, my_func\n", "issue": "Logging output is much more verbose when running with n_jobs > 1 vs n_jobs=1\nAfter #799, I still see lots of output when Reports are created \u2013\u00a0but only if `n_jobs > 1`. For `n_jobs = 1`, things are working as expected.\r\n\r\n```\r\n\u276f mne_bids_pipeline --config=./bids_pipeline_config.py --n_jobs=1\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c Welcome aboard MNE-BIDS-Pipeline! \ud83d\udc4b \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:26\u2502 \ud83d\udcdd Using configuration: ./bids_pipeline_config.py\r\n\u250216:53:26\u2502 \u274c Overriding config.n_jobs = 1\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534 \r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c init/_01_init_derivatives_dir \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:26\u2502 \u23f3\ufe0f Initializing output directories.\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534 done (1s)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c init/_02_find_empty_room \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:26\u2502 \u23e9 Skipping, empty-room data only relevant for MEG \u2026\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534 done (1s)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c preprocessing/_01_data_quality \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:26\u2502 \u23f3\ufe0f sub-005 run-01 Initializing report HDF5 file\r\n\u250216:53:26\u2502 \u23f3\ufe0f sub-005 run-01 Adding original raw data to report\r\n\u250216:53:28\u2502 \u23f3\ufe0f sub-005 run-01 Adding config and sys info to report\r\n```\r\n\r\n```\r\n\u276f mne_bids_pipeline --config=./bids_pipeline_config.py --n_jobs=8\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c Welcome aboard MNE-BIDS-Pipeline! \ud83d\udc4b \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:50\u2502 \ud83d\udcdd Using configuration: ./bids_pipeline_config.py\r\n\u250216:53:50\u2502 \u274c Overriding config.n_jobs = 8\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534 \r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c init/_01_init_derivatives_dir \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:50\u2502 \u2705 Output directories already exist \u2026\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534 done (1s)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c init/_02_find_empty_room \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\n\u250216:53:50\u2502 \u23e9 Skipping, empty-room data only relevant for MEG \u2026\r\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534 done (1s)\r\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c preprocessing/_01_data_quality \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\r\nEmbedding : jquery-3.6.0.min.js\r\nEmbedding : bootstrap.bundle.min.js\r\nEmbedding : bootstrap.min.css\r\nEmbedding : bootstrap-table/bootstrap-table.min.js\r\nEmbedding : bootstrap-table/bootstrap-table.min.css\r\nEmbedding : bootstrap-table/bootstrap-table-copy-rows.min.js\r\nEmbedding : bootstrap-table/bootstrap-table-export.min.js\r\nEmbedding : bootstrap-table/tableExport.min.js\r\nEmbedding : bootstrap-icons/bootstrap-icons.mne.min.css\r\nEmbedding : highlightjs/highlight.min.js\r\nEmbedding : highlightjs/atom-one-dark-reasonable.min.css\r\n\u250216:53:51\u2502 \u23f3\ufe0f sub-005 run-01 Adding original raw data to report\r\n\u250216:53:53\u2502 \u23f3\ufe0f sub-005 run-01 Adding config and sys info to report\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-005 run-01 Saving report: /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-005/eeg/sub-005_task-tasin_report.html\r\nOverwriting existing file.\r\nSaving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-005/eeg/sub-005_task-tasin_report.h5\r\nSaving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-005/eeg/sub-005_task-tasin_report.html\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-102 run-01 Initializing report HDF5 file\r\nSaving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.h5\r\nEmbedding : jquery-3.6.0.min.js\r\nEmbedding : bootstrap.bundle.min.js\r\nEmbedding : bootstrap.min.css\r\nEmbedding : bootstrap-table/bootstrap-table.min.js\r\nEmbedding : bootstrap-table/bootstrap-table.min.css\r\nEmbedding : bootstrap-table/bootstrap-table-copy-rows.min.js\r\nEmbedding : bootstrap-table/bootstrap-table-export.min.js\r\nEmbedding : bootstrap-table/tableExport.min.js\r\nEmbedding : bootstrap-icons/bootstrap-icons.mne.min.css\r\nEmbedding : highlightjs/highlight.min.js\r\nEmbedding : highlightjs/atom-one-dark-reasonable.min.css\r\nEmbedding : jquery-3.6.0.min.js\r\nEmbedding : bootstrap.bundle.min.js\r\nEmbedding : bootstrap.min.css\r\nEmbedding : bootstrap-table/bootstrap-table.min.js\r\nEmbedding : bootstrap-table/bootstrap-table.min.css\r\nEmbedding : bootstrap-table/bootstrap-table-copy-rows.min.js\r\nEmbedding : bootstrap-table/bootstrap-table-export.min.js\r\nEmbedding : bootstrap-table/tableExport.min.js\r\nEmbedding : bootstrap-icons/bootstrap-icons.mne.min.css\r\nEmbedding : highlightjs/highlight.min.js\r\nEmbedding : highlightjs/atom-one-dark-reasonable.min.css\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-102 run-01 Adding original raw data to report\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-005 run-04 Adding original raw data to report\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-005 run-04 Adding config and sys info to report\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-102 run-01 Adding config and sys info to report\r\n\u250216:53:54\u2502 \u23f3\ufe0f sub-102 run-01 Saving report: /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.html\r\nOverwriting existing file.\r\nSaving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.h5\r\nSaving report to : /Users/richardhochenberger/Library/CloudStorage/OneDrive-FirmenichSA/Documents/TasIn/data/derivatives/sub-102/eeg/sub-102_task-tasin_report.html\r\nEmbedding : jquery-3.6.0.min.js\r\nEmbedding : bootstrap.bundle.min.js\r\nEmbedding : bootstrap.min.css\r\nEmbedding : bootstrap-table/bootstrap-table.min.js\r\nEmbedding : bootstrap-table/bootstrap-table.min.css\r\nEmbedding : bootstrap-table/bootstrap-table-copy-rows.min.js\r\nEmbedding : bootstrap-table/bootstrap-table-export.min.js\r\nEmbedding : bootstrap-table/tableExport.min.js\r\nEmbedding : bootstrap-icons/bootstrap-icons.mne.min.css\r\nEmbedding : highlightjs/highlight.min.js\r\nEmbedding : highlightjs/atom-one-dark-reasonable.min.css\r\n```\n", "before_files": [{"content": "\"\"\"Parallelization.\"\"\"\n\nfrom typing import Literal, Callable\nfrom types import SimpleNamespace\n\nimport joblib\n\nfrom ._logging import logger, gen_log_kwargs, _is_testing\n\n\ndef get_n_jobs(*, exec_params: SimpleNamespace, log_override: bool = False) -> int:\n    n_jobs = exec_params.n_jobs\n    if n_jobs < 0:\n        n_cores = joblib.cpu_count()\n        n_jobs = min(n_cores + n_jobs + 1, n_cores)\n\n    # Shim to allow overriding n_jobs for specific steps\n    if _is_testing() and hasattr(exec_params, \"_n_jobs\"):\n        from ._run import _get_step_path, _short_step_path\n\n        step_path = _short_step_path(_get_step_path())\n        orig_n_jobs = n_jobs\n        n_jobs = exec_params._n_jobs.get(step_path, n_jobs)\n        if log_override and n_jobs != orig_n_jobs:\n            msg = f\"Overriding n_jobs: {orig_n_jobs}\u2192{n_jobs}\"\n            logger.info(**gen_log_kwargs(message=msg, emoji=\"override\"))\n    return n_jobs\n\n\ndask_client = None\n\n\ndef setup_dask_client(*, exec_params: SimpleNamespace) -> None:\n    global dask_client\n\n    import dask\n    from dask.distributed import Client\n\n    if dask_client is not None:\n        return\n\n    n_workers = get_n_jobs(exec_params=exec_params)\n    msg = f\"Dask initializing with {n_workers} workers \u2026\"\n    logger.info(**gen_log_kwargs(message=msg, emoji=\"\ud83d\udc7e\"))\n\n    if exec_params.dask_temp_dir is None:\n        this_dask_temp_dir = exec_params.deriv_root / \".dask-worker-space\"\n    else:\n        this_dask_temp_dir = exec_params.dask_temp_dir\n\n    msg = f\"Dask temporary directory: {this_dask_temp_dir}\"\n    logger.info(**gen_log_kwargs(message=msg, emoji=\"\ud83d\udcc2\"))\n    dask.config.set(\n        {\n            \"temporary-directory\": this_dask_temp_dir,\n            \"distributed.worker.memory.pause\": 0.8,\n            # fraction of memory that can be utilized before the nanny\n            # process will terminate the worker\n            \"distributed.worker.memory.terminate\": 1.0,\n            # TODO spilling to disk currently doesn't work reliably for us,\n            # as Dask cannot spill \"unmanaged\" memory \u2013\u00a0and most of what we\n            # see currently is, in fact, \"unmanaged\". Needs thorough\n            # investigation.\n            \"distributed.worker.memory.spill\": False,\n        }\n    )\n    client = Client(  # noqa: F841\n        memory_limit=exec_params.dask_worker_memory_limit,\n        n_workers=n_workers,\n        threads_per_worker=1,\n        name=\"mne-bids-pipeline\",\n    )\n    client.auto_restart = False  # don't restart killed workers\n\n    dashboard_url = client.dashboard_link\n    msg = \"Dask client dashboard: \" f\"[link={dashboard_url}]{dashboard_url}[/link]\"\n    logger.info(**gen_log_kwargs(message=msg, emoji=\"\ud83c\udf0e\"))\n\n    if exec_params.dask_open_dashboard:\n        import webbrowser\n\n        webbrowser.open(url=dashboard_url, autoraise=True)\n\n    # Update global variable\n    dask_client = client\n\n\ndef get_parallel_backend_name(\n    *,\n    exec_params: SimpleNamespace,\n) -> Literal[\"dask\", \"loky\"]:\n    if (\n        exec_params.parallel_backend == \"loky\"\n        or get_n_jobs(exec_params=exec_params) == 1\n    ):\n        backend = \"loky\"\n    elif exec_params.parallel_backend == \"dask\":\n        # Disable interactive plotting backend\n        import matplotlib\n\n        matplotlib.use(\"Agg\")\n        backend = \"dask\"\n    else:\n        # TODO: Move to value validation step\n        raise ValueError(f\"Unknown parallel backend: {exec_params.parallel_backend}\")\n\n    return backend\n\n\ndef get_parallel_backend(exec_params: SimpleNamespace) -> joblib.parallel_backend:\n    import joblib\n\n    backend = get_parallel_backend_name(exec_params=exec_params)\n    kwargs = {\n        \"n_jobs\": get_n_jobs(\n            exec_params=exec_params,\n            log_override=True,\n        )\n    }\n\n    if backend == \"loky\":\n        kwargs[\"inner_max_num_threads\"] = 1\n    else:\n        setup_dask_client(exec_params=exec_params)\n\n    return joblib.parallel_backend(backend, **kwargs)\n\n\ndef parallel_func(func: Callable, *, exec_params: SimpleNamespace):\n    if get_parallel_backend_name(exec_params=exec_params) == \"loky\":\n        if get_n_jobs(exec_params=exec_params) == 1:\n            my_func = func\n            parallel = list\n        else:\n            from joblib import Parallel, delayed\n\n            parallel = Parallel()\n            my_func = delayed(func)\n    else:  # Dask\n        from joblib import Parallel, delayed\n\n        parallel = Parallel()\n        my_func = delayed(func)\n\n    return parallel, my_func\n", "path": "mne_bids_pipeline/_parallel.py"}], "after_files": [{"content": "\"\"\"Parallelization.\"\"\"\n\nfrom typing import Literal, Callable\nfrom types import SimpleNamespace\n\nimport joblib\nfrom mne.utils import use_log_level, logger as mne_logger\n\nfrom ._logging import logger, gen_log_kwargs, _is_testing\n\n\ndef get_n_jobs(*, exec_params: SimpleNamespace, log_override: bool = False) -> int:\n    n_jobs = exec_params.n_jobs\n    if n_jobs < 0:\n        n_cores = joblib.cpu_count()\n        n_jobs = min(n_cores + n_jobs + 1, n_cores)\n\n    # Shim to allow overriding n_jobs for specific steps\n    if _is_testing() and hasattr(exec_params, \"_n_jobs\"):\n        from ._run import _get_step_path, _short_step_path\n\n        step_path = _short_step_path(_get_step_path())\n        orig_n_jobs = n_jobs\n        n_jobs = exec_params._n_jobs.get(step_path, n_jobs)\n        if log_override and n_jobs != orig_n_jobs:\n            msg = f\"Overriding n_jobs: {orig_n_jobs}\u2192{n_jobs}\"\n            logger.info(**gen_log_kwargs(message=msg, emoji=\"override\"))\n    return n_jobs\n\n\ndask_client = None\n\n\ndef setup_dask_client(*, exec_params: SimpleNamespace) -> None:\n    global dask_client\n\n    import dask\n    from dask.distributed import Client\n\n    if dask_client is not None:\n        return\n\n    n_workers = get_n_jobs(exec_params=exec_params)\n    msg = f\"Dask initializing with {n_workers} workers \u2026\"\n    logger.info(**gen_log_kwargs(message=msg, emoji=\"\ud83d\udc7e\"))\n\n    if exec_params.dask_temp_dir is None:\n        this_dask_temp_dir = exec_params.deriv_root / \".dask-worker-space\"\n    else:\n        this_dask_temp_dir = exec_params.dask_temp_dir\n\n    msg = f\"Dask temporary directory: {this_dask_temp_dir}\"\n    logger.info(**gen_log_kwargs(message=msg, emoji=\"\ud83d\udcc2\"))\n    dask.config.set(\n        {\n            \"temporary-directory\": this_dask_temp_dir,\n            \"distributed.worker.memory.pause\": 0.8,\n            # fraction of memory that can be utilized before the nanny\n            # process will terminate the worker\n            \"distributed.worker.memory.terminate\": 1.0,\n            # TODO spilling to disk currently doesn't work reliably for us,\n            # as Dask cannot spill \"unmanaged\" memory \u2013\u00a0and most of what we\n            # see currently is, in fact, \"unmanaged\". Needs thorough\n            # investigation.\n            \"distributed.worker.memory.spill\": False,\n        }\n    )\n    client = Client(  # noqa: F841\n        memory_limit=exec_params.dask_worker_memory_limit,\n        n_workers=n_workers,\n        threads_per_worker=1,\n        name=\"mne-bids-pipeline\",\n    )\n    client.auto_restart = False  # don't restart killed workers\n\n    dashboard_url = client.dashboard_link\n    msg = \"Dask client dashboard: \" f\"[link={dashboard_url}]{dashboard_url}[/link]\"\n    logger.info(**gen_log_kwargs(message=msg, emoji=\"\ud83c\udf0e\"))\n\n    if exec_params.dask_open_dashboard:\n        import webbrowser\n\n        webbrowser.open(url=dashboard_url, autoraise=True)\n\n    # Update global variable\n    dask_client = client\n\n\ndef get_parallel_backend_name(\n    *,\n    exec_params: SimpleNamespace,\n) -> Literal[\"dask\", \"loky\"]:\n    if (\n        exec_params.parallel_backend == \"loky\"\n        or get_n_jobs(exec_params=exec_params) == 1\n    ):\n        backend = \"loky\"\n    elif exec_params.parallel_backend == \"dask\":\n        # Disable interactive plotting backend\n        import matplotlib\n\n        matplotlib.use(\"Agg\")\n        backend = \"dask\"\n    else:\n        # TODO: Move to value validation step\n        raise ValueError(f\"Unknown parallel backend: {exec_params.parallel_backend}\")\n\n    return backend\n\n\ndef get_parallel_backend(exec_params: SimpleNamespace) -> joblib.parallel_backend:\n    import joblib\n\n    backend = get_parallel_backend_name(exec_params=exec_params)\n    kwargs = {\n        \"n_jobs\": get_n_jobs(\n            exec_params=exec_params,\n            log_override=True,\n        )\n    }\n\n    if backend == \"loky\":\n        kwargs[\"inner_max_num_threads\"] = 1\n    else:\n        setup_dask_client(exec_params=exec_params)\n\n    return joblib.parallel_backend(backend, **kwargs)\n\n\ndef parallel_func(func: Callable, *, exec_params: SimpleNamespace):\n    if (\n        get_parallel_backend_name(exec_params=exec_params) == \"loky\"\n        and get_n_jobs(exec_params=exec_params) == 1\n    ):\n        my_func = func\n        parallel = list\n    else:  # Dask or n_jobs > 1\n        from joblib import Parallel, delayed\n\n        parallel = Parallel()\n\n        def run_verbose(*args, verbose=mne_logger.level, **kwargs):\n            with use_log_level(verbose=verbose):\n                return func(*args, **kwargs)\n\n        my_func = delayed(run_verbose)\n\n    return parallel, my_func\n", "path": "mne_bids_pipeline/_parallel.py"}]}