Make `pyarrow` optional
CONTRIBUTOR
The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)
2022-09-16T10:03:47Z
https://github.com/huggingface/datasets/issues/4980#issuecomment-1249176233
https://api.github.com/repos/huggingface/datasets/issues/4980
IC_kwDODunzps5KdOqp
https://api.github.com/repos/huggingface/datasets/issues/comments/1249176233/reactions
2022-09-16T10:03:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1249176233
https://avatars.githubusercontent.com/u/15098095?v=4
https://api.github.com/users/dconathan/events{/privacy}
https://api.github.com/users/dconathan/followers
https://api.github.com/users/dconathan/following{/other_user}
https://api.github.com/users/dconathan/gists{/gist_id}
https://github.com/dconathan
dconathan
MDQ6VXNlcjE1MDk4MDk1
https://api.github.com/users/dconathan/orgs
https://api.github.com/users/dconathan/received_events
https://api.github.com/users/dconathan/repos
https://api.github.com/users/dconathan/starred{/owner}{/repo}
https://api.github.com/users/dconathan/subscriptions
User
https://api.github.com/users/dconathan
public
MEMBER
Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached. 
2022-09-16T14:07:18Z
https://github.com/huggingface/datasets/issues/4980#issuecomment-1249409522
https://api.github.com/repos/huggingface/datasets/issues/4980
IC_kwDODunzps5KeHny
https://api.github.com/repos/huggingface/datasets/issues/comments/1249409522/reactions
2022-09-16T14:07:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1249409522
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.

2022-09-16T17:23:36Z
https://github.com/huggingface/datasets/issues/4980#issuecomment-1249606729
https://api.github.com/repos/huggingface/datasets/issues/4980
IC_kwDODunzps5Ke3xJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1249606729/reactions
2022-09-16T17:23:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1249606729
https://avatars.githubusercontent.com/u/240344?v=4
https://api.github.com/users/KOLANICH/events{/privacy}
https://api.github.com/users/KOLANICH/followers
https://api.github.com/users/KOLANICH/following{/other_user}
https://api.github.com/users/KOLANICH/gists{/gist_id}
https://github.com/KOLANICH
KOLANICH
MDQ6VXNlcjI0MDM0NA==
https://api.github.com/users/KOLANICH/orgs
https://api.github.com/users/KOLANICH/received_events
https://api.github.com/users/KOLANICH/repos
https://api.github.com/users/KOLANICH/starred{/owner}{/repo}
https://api.github.com/users/KOLANICH/subscriptions
User
https://api.github.com/users/KOLANICH
public
**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.
