`_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`
MEMBER
This is maybe related to python 3.10, do you think you could try on 3.8 ?

In the meantime we'll keep improving the support for 3.10. Let me add a dedicated CI
2022-10-04T09:29:42Z
https://github.com/huggingface/datasets/issues/5061#issuecomment-1266663968
https://api.github.com/repos/huggingface/datasets/issues/5061
IC_kwDODunzps5Lf8Ig
https://api.github.com/repos/huggingface/datasets/issues/comments/1266663968/reactions
2022-10-04T09:29:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1266663968
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I did some binary search and seems like the root cause is either `multiprocess` or `dill`. python 3.10 is fine. Specifically:
- `multiprocess==0.70.12.2, dill==0.3.4`: works
- `multiprocess==0.70.12.2, dill==0.3.5.1`: doesn't work
- `multiprocess==0.70.13, dill==0.3.5.1`: doesn't work
- `multiprocess==0.70.13, dill==0.3.4`: can't test, `multiprocess==0.70.13` requires `dill>=0.3.5.1`

I will pin their versions on my end. I don't have enough knowledge of how python multiprocessing works to debug this, but ideally there could be a fix. It's also possible that I'm doing something wrong in my code, but again the `.name` of the logger that failed to pickle is `datasets.fingerprint`, which I'm not using directly.
2022-10-04T15:48:22Z
https://github.com/huggingface/datasets/issues/5061#issuecomment-1267211971
https://api.github.com/repos/huggingface/datasets/issues/5061
IC_kwDODunzps5LiB7D
https://api.github.com/repos/huggingface/datasets/issues/comments/1267211971/reactions
2022-10-04T15:48:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1267211971
https://avatars.githubusercontent.com/u/11954789?v=4
https://api.github.com/users/ZhaofengWu/events{/privacy}
https://api.github.com/users/ZhaofengWu/followers
https://api.github.com/users/ZhaofengWu/following{/other_user}
https://api.github.com/users/ZhaofengWu/gists{/gist_id}
https://github.com/ZhaofengWu
ZhaofengWu
MDQ6VXNlcjExOTU0Nzg5
https://api.github.com/users/ZhaofengWu/orgs
https://api.github.com/users/ZhaofengWu/received_events
https://api.github.com/users/ZhaofengWu/repos
https://api.github.com/users/ZhaofengWu/starred{/owner}{/repo}
https://api.github.com/users/ZhaofengWu/subscriptions
User
https://api.github.com/users/ZhaofengWu
public
MEMBER
Do you know which logger fails at being pickled ?
2022-10-14T15:18:36Z
https://github.com/huggingface/datasets/issues/5061#issuecomment-1279145039
https://api.github.com/repos/huggingface/datasets/issues/5061
IC_kwDODunzps5MPjRP
https://api.github.com/repos/huggingface/datasets/issues/comments/1279145039/reactions
2022-10-14T15:18:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279145039
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm not 100% sure how to figure it out -- the stack trace above doesn't clearly give me a place where I can print out who owns the logger, etc. I only found out its `.name` is `datasets.fingerprint` by printing right before
```
  File ".../logging/__init__.py", line 1774, in __reduce__
    raise pickle.PicklingError('logger cannot be pickled')
```
If you have any idea on how to find it out, please let me know.
2022-10-14T15:50:31Z
https://github.com/huggingface/datasets/issues/5061#issuecomment-1279179143
https://api.github.com/repos/huggingface/datasets/issues/5061
IC_kwDODunzps5MPrmH
https://api.github.com/repos/huggingface/datasets/issues/comments/1279179143/reactions
2022-10-14T15:50:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279179143
https://avatars.githubusercontent.com/u/11954789?v=4
https://api.github.com/users/ZhaofengWu/events{/privacy}
https://api.github.com/users/ZhaofengWu/followers
https://api.github.com/users/ZhaofengWu/following{/other_user}
https://api.github.com/users/ZhaofengWu/gists{/gist_id}
https://github.com/ZhaofengWu
ZhaofengWu
MDQ6VXNlcjExOTU0Nzg5
https://api.github.com/users/ZhaofengWu/orgs
https://api.github.com/users/ZhaofengWu/received_events
https://api.github.com/users/ZhaofengWu/repos
https://api.github.com/users/ZhaofengWu/starred{/owner}{/repo}
https://api.github.com/users/ZhaofengWu/subscriptions
User
https://api.github.com/users/ZhaofengWu
public
MEMBER
Ok I see, not sure why it triggers this error though, in `logging.py` the code is

https://github.com/python/cpython/blob/c9da063e32725a66495e4047b8a5ed13e72d9e8e/Lib/logging/__init__.py#L1769-L1775

and on my side it works on 3.10 with dill 0.3.5.1 and multiprocess 0.70.13
```python
>>> datasets.fingerprint.logger.__reduce__()                                                                                                         
(<function logging.getLogger(name=None)>, ('datasets.fingerprint',))
```
Could you try to run this code ?

Are you in an environment where the loggers are instantiated differently ? Can you check the source code of `logging.Logger.__reduce__` in `".../logging/__init__.py", line 1774` ?
2022-10-14T16:44:54Z
https://github.com/huggingface/datasets/issues/5061#issuecomment-1279238014
https://api.github.com/repos/huggingface/datasets/issues/5061
IC_kwDODunzps5MP59-
https://api.github.com/repos/huggingface/datasets/issues/comments/1279238014/reactions
2022-10-14T16:44:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1279238014
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Closing due to inactivity.
2023-07-21T14:43:34Z
https://github.com/huggingface/datasets/issues/5061#issuecomment-1645704437
https://api.github.com/repos/huggingface/datasets/issues/5061
IC_kwDODunzps5iF3T1
https://api.github.com/repos/huggingface/datasets/issues/comments/1645704437/reactions
2023-07-21T14:43:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1645704437
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
## Describe the bug
When I `map` with multiple processes, this error occurs. The `.name` of the `logger` that fails to pickle in the final line is `datasets.fingerprint`.
```
  File "~/project/dataset.py", line 204, in <dictcomp>
    split: dataset.map(
  File ".../site-packages/datasets/arrow_dataset.py", line 2489, in map
    transformed_shards[index] = async_result.get()
  File ".../site-packages/multiprocess/pool.py", line 771, in get
    raise self._value
  File ".../site-packages/multiprocess/pool.py", line 537, in _handle_tasks
    put(task)
  File ".../site-packages/multiprocess/connection.py", line 214, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File ".../site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File ".../site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File ".../pickle.py", line 487, in dump
    self.save(obj)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../pickle.py", line 902, in save_tuple
    save(element)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File ".../site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File ".../pickle.py", line 717, in save_reduce
    save(state)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../pickle.py", line 887, in save_tuple
    save(element)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File ".../pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File ".../pickle.py", line 998, in _batch_setitems
    save(v)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File ".../site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File ".../pickle.py", line 717, in save_reduce
    save(state)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../pickle.py", line 887, in save_tuple
    save(element)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File ".../pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File ".../pickle.py", line 998, in _batch_setitems
    save(v)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File ".../site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File ".../pickle.py", line 998, in _batch_setitems
    save(v)
  File ".../pickle.py", line 578, in save
    rv = reduce(self.proto)
  File ".../logging/__init__.py", line 1774, in __reduce__
    raise pickle.PicklingError('logger cannot be pickled')
_pickle.PicklingError: logger cannot be pickled
```

## Steps to reproduce the bug
Sorry I failed to have a minimal reproducible example, but the offending line on my end is
```python
dataset.map(
    lambda examples: self.tokenize(examples),  # this doesn't matter, lambda e: [1] * len(...) also breaks. In fact I'm pretty sure it breaks before executing this lambda
    batched=True,
    num_proc=4,
)
```
This does work when `num_proc=1`, so it's likely a multiprocessing thing.

## Expected results
`map` succeeds

## Actual results
The error trace above.

## Environment info
- `datasets` version: 1.16.1 and 2.5.1 both failed
- Platform: Ubuntu 20.04.4 LTS
- Python version: 3.10.4
- PyArrow version: 9.0.0
