Support streaming mlsum dataset
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-06-27T07:44:59Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1167001280
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5FjwbA
https://api.github.com/repos/huggingface/datasets/issues/comments/1167001280/reactions
2022-07-21T12:53:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1167001280
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
After unpinning `s3fs` and pinning `fsspec[http]>=2021.11.1`, the CI installs
- `fsspec-2022.1.0`
- `s3fs-0.5.1`

and raises the following error:
```
 ImportError while loading conftest '/home/runner/work/datasets/datasets/tests/conftest.py'.
tests/conftest.py:13: in <module>
    import datasets
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/__init__.py:37: in <module>
    from .arrow_dataset import Dataset
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/arrow_dataset.py:62: in <module>
    from .arrow_reader import ArrowReader
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/arrow_reader.py:29: in <module>
    from .download.download_config import DownloadConfig
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/download/__init__.py:10: in <module>
    from .streaming_download_manager import StreamingDownloadManager
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/download/streaming_download_manager.py:20: in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/filesystems/__init__.py:13: in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datasets/filesystems/s3filesystem.py:1: in <module>
    import s3fs
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/s3fs/__init__.py:1: in <module>
    from .core import S3FileSystem, S3File
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/s3fs/core.py:12: in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
E   ImportError: cannot import name 'maybe_sync'
```

The installed `s3fs` version is too old. What about pinning a min version?
2022-07-21T09:47:09Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1191277447
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5HAXOH
https://api.github.com/repos/huggingface/datasets/issues/comments/1191277447/reactions
2022-07-21T09:53:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191277447
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Maybe you can try setting the same minimum version as fsspec ? `s3fs>=2021.11.1`
2022-07-21T10:03:30Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1191293734
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5HAbMm
https://api.github.com/repos/huggingface/datasets/issues/comments/1191293734/reactions
2022-07-21T10:03:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191293734
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Yes, I have checked that they both require to have the same version. 

The issue then was coming from aiobotocore, boto3, botocore. I have changed them from strict to min version requirements.
> s3fs 2021.11.1 depends on aiobotocore~=2.0.1
2022-07-21T10:21:22Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1191311168
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5HAfdA
https://api.github.com/repos/huggingface/datasets/issues/comments/1191311168/reactions
2022-07-21T10:23:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191311168
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I have updated all min versions so that they are compatible one with each other. I'm pushing again...
2022-07-21T10:41:08Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1191329549
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5HAj8N
https://api.github.com/repos/huggingface/datasets/issues/comments/1191329549/reactions
2022-07-21T10:41:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191329549
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thanks !
2022-07-21T13:05:34Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1191461546
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5HBEKq
https://api.github.com/repos/huggingface/datasets/issues/comments/1191461546/reactions
2022-07-21T13:05:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191461546
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Nice!
2022-07-21T13:37:29Z
https://github.com/huggingface/datasets/pull/4574#issuecomment-1191496509
https://api.github.com/repos/huggingface/datasets/issues/4574
IC_kwDODunzps5HBMs9
https://api.github.com/repos/huggingface/datasets/issues/comments/1191496509/reactions
2022-07-21T13:37:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191496509
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
Support streaming mlsum dataset.

This PR:
- pins `fsspec` min version with fixed BlockSizeError: `fsspec[http]>=2021.11.1`
  - https://github.com/fsspec/filesystem_spec/pull/830
- unpins `s3fs==2021.08.1` to align it with `fsspec` requirement: `s3fs>=2021.11.1`
  > s3fs 2021.8.1 requires fsspec==2021.08.1
  - see discussion: https://github.com/huggingface/datasets/pull/2858/files#r700027326
- updates the following requirements to be compatible with the previous ones and one with each other:
  - `aiobotocore==1.4.2` to `aiobotocore>=2.0.1` (required by s3fs>=2021.11.1)
  - `boto3==1.17.106` to `boto3>=1.19.8` (to be compatible with aiobotocore>=2.0.1)
  - `botocore==1.20.106` to `botocore>=1.22.8` (to be compatible with aiobotocore and boto3)

Fix #4572.