Compatibility issue between s3fs, fsspec, and datasets
COLLABORATOR
Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.
2024-04-03T13:44:27Z
https://github.com/huggingface/datasets/issues/6765#issuecomment-2034658640
https://api.github.com/repos/huggingface/datasets/issues/6765
IC_kwDODunzps55Rm1Q
https://api.github.com/repos/huggingface/datasets/issues/comments/2034658640/reactions
2024-04-03T13:44:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2034658640
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

Thanks so much! My inexperience with pip is showing 😆  🙈 
2024-04-03T14:33:12Z
https://github.com/huggingface/datasets/issues/6765#issuecomment-2034797488
https://api.github.com/repos/huggingface/datasets/issues/6765
IC_kwDODunzps55SIuw
https://api.github.com/repos/huggingface/datasets/issues/comments/2034797488/reactions
2024-04-03T14:33:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2034797488
https://avatars.githubusercontent.com/u/33383515?v=4
https://api.github.com/users/njbrake/events{/privacy}
https://api.github.com/users/njbrake/followers
https://api.github.com/users/njbrake/following{/other_user}
https://api.github.com/users/njbrake/gists{/gist_id}
https://github.com/njbrake
njbrake
MDQ6VXNlcjMzMzgzNTE1
https://api.github.com/users/njbrake/orgs
https://api.github.com/users/njbrake/received_events
https://api.github.com/users/njbrake/repos
https://api.github.com/users/njbrake/starred{/owner}{/repo}
https://api.github.com/users/njbrake/subscriptions
User
https://api.github.com/users/njbrake
public
NONE
> Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

you are awesome bro
2024-05-05T13:37:13Z
https://github.com/huggingface/datasets/issues/6765#issuecomment-2094813467
https://api.github.com/repos/huggingface/datasets/issues/6765
IC_kwDODunzps583FEb
https://api.github.com/repos/huggingface/datasets/issues/comments/2094813467/reactions
2024-05-05T13:37:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2094813467
https://avatars.githubusercontent.com/u/56914156?v=4
https://api.github.com/users/poult-lab/events{/privacy}
https://api.github.com/users/poult-lab/followers
https://api.github.com/users/poult-lab/following{/other_user}
https://api.github.com/users/poult-lab/gists{/gist_id}
https://github.com/poult-lab
poult-lab
MDQ6VXNlcjU2OTE0MTU2
https://api.github.com/users/poult-lab/orgs
https://api.github.com/users/poult-lab/received_events
https://api.github.com/users/poult-lab/repos
https://api.github.com/users/poult-lab/starred{/owner}{/repo}
https://api.github.com/users/poult-lab/subscriptions
User
https://api.github.com/users/poult-lab
public
NONE
Hey, the suggestion by @mariosasko unfortunately only address this issue via pip. The original message was about poetry and I am still facing a dependency conflict with that.

The following command complains first about `fsspec` (`... no versions of fsspec match ...`) and then I get an error.

Command:
`poetry add datasets s3fs` 

Error: 
` ... your project ... depends on both datasets (^3.1.0) and s3fs (^2024.10.0), version solving failed`

Installing first `s3fs` and then the rest of the huggingface libraries, like `datasets`, also did not help.
2024-11-12T14:50:46Z
https://github.com/huggingface/datasets/issues/6765#issuecomment-2470742779
https://api.github.com/repos/huggingface/datasets/issues/6765
IC_kwDODunzps6TRIr7
https://api.github.com/repos/huggingface/datasets/issues/comments/2470742779/reactions
2024-11-12T14:50:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2470742779
https://avatars.githubusercontent.com/u/64789450?v=4
https://api.github.com/users/Konstantina-Lazaridou/events{/privacy}
https://api.github.com/users/Konstantina-Lazaridou/followers
https://api.github.com/users/Konstantina-Lazaridou/following{/other_user}
https://api.github.com/users/Konstantina-Lazaridou/gists{/gist_id}
https://github.com/Konstantina-Lazaridou
Konstantina-Lazaridou
MDQ6VXNlcjY0Nzg5NDUw
https://api.github.com/users/Konstantina-Lazaridou/orgs
https://api.github.com/users/Konstantina-Lazaridou/received_events
https://api.github.com/users/Konstantina-Lazaridou/repos
https://api.github.com/users/Konstantina-Lazaridou/starred{/owner}{/repo}
https://api.github.com/users/Konstantina-Lazaridou/subscriptions
User
https://api.github.com/users/Konstantina-Lazaridou
public
### Describe the bug

Here is the full error stack when installing:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you have fsspec 2024.3.1 which is incompatible.
Successfully installed aiobotocore-2.12.1 aioitertools-0.11.0 botocore-1.34.51 fsspec-2024.3.1 jmespath-1.0.1 s3fs-2024.3.1 urllib3-2.0.7 wrapt-1.16.0
```
When I install with pip, pip allows this error to exist while still installing s3fs, but this error breaks poetry, since poetry will refuse to install s3fs because of the dependency conflict.

Maybe I'm missing something so maybe it's not a bug but some mistake on my end? Any input would be helpful. Thanks!

### Steps to reproduce the bug

1. conda create -n tmp python=3.10 -y
2. conda activate tmp
3. pip install datasets
4. pip install s3fs

### Expected behavior

I would expect there to be no error.

### Environment info

MacOS (ARM), Python3.10, conda 23.11.0. 