Support dill 0.3.6
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-10-26T08:29:22Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1291680754
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5M_Xvy
https://api.github.com/repos/huggingface/datasets/issues/comments/1291680754/reactions
2022-10-28T05:41:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1291680754
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
I think it hasn't been merged ? https://github.com/uqfoundation/dill/pull/501

Though I can see that the CI is green because it uses dill 0.3.1.1 - we should probably fix the dill version in both CIs:
- use 0.3.1.1 for the CI with the minimum requirements
- use latest for the CI with the latest requirements
2022-10-26T09:33:43Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1291763334
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5M_r6G
https://api.github.com/repos/huggingface/datasets/issues/comments/1291763334/reactions
2022-10-26T09:33:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1291763334
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I have noticed our CI uses `dill-0.3.1.1`, so not really testing dill 0.3.6...
2022-10-26T09:38:11Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1291768094
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5M_tEe
https://api.github.com/repos/huggingface/datasets/issues/comments/1291768094/reactions
2022-10-26T09:38:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1291768094
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
The dill version in our CI is due to `apache-beam`...
2022-10-26T09:39:21Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1291769294
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5M_tXO
https://api.github.com/repos/huggingface/datasets/issues/comments/1291769294/reactions
2022-10-26T09:40:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1291769294
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I've tested locally: we need a specific fix for 0.3.6 (different from the previous ones)...
2022-10-26T09:39:57Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1291769923
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5M_thD
https://api.github.com/repos/huggingface/datasets/issues/comments/1291769923/reactions
2022-10-26T09:40:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1291769923
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I think we can force the version of dill to be whatever we want in the CI - no matter what beam says. The alternative would be to run beam tests separately but it's more work
2022-10-26T13:26:45Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1292034348
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5NAuEs
https://api.github.com/repos/huggingface/datasets/issues/comments/1292034348/reactions
2022-10-26T13:26:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1292034348
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq I tried the easiest solution: force dill==0.3.6 ignoring the requirement of apache-beam. But it doesn't work:
- For example, for `tests/test_builder.py::test_beam_based_builder_download_and_prepare_as_parquet`:
```
    @dill.dill.register(dill.dill.ModuleType)
    def save_module(pickler, obj):
      if dill.dill.is_dill(pickler) and obj is pickler._main:
        return old_save_module(pickler, obj)
      else:
>       dill.dill.log.info('M2: %s' % obj)
E       AttributeError: module 'dill._dill' has no attribute 'log'

venv/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:170: AttributeError
```
  - Apache Beam registers some dill functions (`save_module`) which are incompatible with dill 0.3.6 (in 0.3.6 'dill._dill' has no attribute 'log' but 'logger')
  - This has an impact in CI tests using either Apache Beam or `multiprocess` (even without using Apache Beam!):
```
FAILED tests/test_beam.py::BeamBuilderTest::test_download_and_prepare - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_beam.py::BeamBuilderTest::test_nested_features - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_multiprocessing_in_memory - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_multiprocessing_on_disk - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_builder.py::test_beam_based_download_and_prepare - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_in_memory - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_builder.py::test_beam_based_as_dataset - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_multiprocessing_in_memory - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_multiprocessing_on_disk - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_builder.py::test_beam_based_builder_download_and_prepare_as_parquet - AttributeError: module 'dill._dill' has no attribute 'log'
```

I guess we should implement the other option: run beam tests separately.

I'm opening another PR for the CI refactoring.
2022-10-27T06:58:43Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1293079055
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5NEtIP
https://api.github.com/repos/huggingface/datasets/issues/comments/1293079055/reactions
2022-10-27T07:58:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1293079055
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Ah crap >< maybe only install apache_beam for the "minimum requirements" CI
2022-10-27T08:19:33Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1293163825
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5NFB0x
https://api.github.com/repos/huggingface/datasets/issues/comments/1293163825/reactions
2022-10-27T08:19:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1293163825
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq if we install apache-beam only in the "minimum requirements" CI, then this other PR should be merged first:
- #5168 

Otherwise, our CI for "latest" will fail because it will try to run the beam tests (because PyTorch is installed but indeed apache-beam is not installed).
2022-10-27T09:00:55Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1293212753
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5NFNxR
https://api.github.com/repos/huggingface/datasets/issues/comments/1293212753/reactions
2022-10-27T09:01:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1293212753
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
One of the test is failing because we set 
```python
# google colab doesn't allow to pickle loggers
# so we want to make sure each tests passes without pickling the logger
def reduce_ex(self):
    raise pickle.PicklingError()

datasets.arrow_dataset.logger.__reduce_ex__ = reduce_ex
```
in `test_arrow_dataset.py` to avoid pickling the logger because it used to fail on google colab.

Now pickling the logger seems to be working on google colab again - so you can remove it, and it should fix some tests
2022-10-27T12:22:23Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1293445536
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5NGGmg
https://api.github.com/repos/huggingface/datasets/issues/comments/1293445536/reactions
2022-10-27T12:23:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1293445536
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
For the other 2 errors:
- FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_in_memory - _pickle.PicklingError: Can't pickle <class 'unittest.mock.MagicMock'>: it's not the same object as unittest.mock.MagicMock
- FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - _pickle.PicklingError: Can't pickle <class 'unittest.mock.MagicMock'>: it's not the same object as unittest.mock.MagicMock

I have implemented a pickable MagicMock.
2022-10-27T13:50:28Z
https://github.com/huggingface/datasets/pull/5166#issuecomment-1293556920
https://api.github.com/repos/huggingface/datasets/issues/5166
IC_kwDODunzps5NGhy4
https://api.github.com/repos/huggingface/datasets/issues/comments/1293556920/reactions
2022-10-27T13:50:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1293556920
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
This PR:
- ~~Unpins dill to allow installing dill>=0.3.6~~
- ~~Removes the fix on dill for >=0.3.6 because they implemented a deterministic mode (to be confirmed by @anivegesana)~~
- Pins dill<0.3.7 to allow latest dill 0.3.6
- Implements a fix for dill `save_function` for dill 0.3.6
- Additionally had to implement a fix for dill `save_code` and `_save_regex` for dill 0.3.6
- Fixes the CI so that the latest dill version is tested (besides the minimum 0.3.1.1 required by apache-beam 2.42.0)

Fix #5162.