How to disable making arrow tables in load_dataset ?
MEMBER
Hi ! We plan to add streaming features in the future.

This should allow to load a dataset instantaneously without generating the arrow table. The trade-off is that accessing examples from a streaming dataset must be done in an iterative way, and with an additional (but hopefully minor) overhead.
What do you think about this ?

If you have ideas or suggestions of what you expect from such features as a user, feel free to share them, this is really valuable to us !
2021-03-24T10:15:26Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-805677181
https://api.github.com/repos/huggingface/datasets/issues/2092
MDEyOklzc3VlQ29tbWVudDgwNTY3NzE4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/805677181/reactions
2021-03-24T10:15:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/805677181
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
People mainly want this feature either because it takes too much time too make arrow tables, or they occupy too much memory on the disk. I think both the problem can be solved if we provide arrow tables themselves on datasets hub. Can we do this currently @lhoestq ? 

2021-03-25T12:41:14Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-806668350
https://api.github.com/repos/huggingface/datasets/issues/2092
MDEyOklzc3VlQ29tbWVudDgwNjY2ODM1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/806668350/reactions
2021-03-25T12:41:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/806668350
https://avatars.githubusercontent.com/u/48825663?v=4
https://api.github.com/users/Jeevesh8/events{/privacy}
https://api.github.com/users/Jeevesh8/followers
https://api.github.com/users/Jeevesh8/following{/other_user}
https://api.github.com/users/Jeevesh8/gists{/gist_id}
https://github.com/Jeevesh8
Jeevesh8
MDQ6VXNlcjQ4ODI1NjYz
https://api.github.com/users/Jeevesh8/orgs
https://api.github.com/users/Jeevesh8/received_events
https://api.github.com/users/Jeevesh8/repos
https://api.github.com/users/Jeevesh8/starred{/owner}{/repo}
https://api.github.com/users/Jeevesh8/subscriptions
User
https://api.github.com/users/Jeevesh8
public
NONE
@lhoestq I think the ```try_from_hf_gcs``` provide the same functionality. What all datasets are available on HF GCS? Are all the datasets on huggingFace datasets hub are made available on GCS, automatically?
2021-03-25T13:44:24Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-806768044
https://api.github.com/repos/huggingface/datasets/issues/2092
MDEyOklzc3VlQ29tbWVudDgwNjc2ODA0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/806768044/reactions
2021-03-25T13:44:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/806768044
https://avatars.githubusercontent.com/u/48825663?v=4
https://api.github.com/users/Jeevesh8/events{/privacy}
https://api.github.com/users/Jeevesh8/followers
https://api.github.com/users/Jeevesh8/following{/other_user}
https://api.github.com/users/Jeevesh8/gists{/gist_id}
https://github.com/Jeevesh8
Jeevesh8
MDQ6VXNlcjQ4ODI1NjYz
https://api.github.com/users/Jeevesh8/orgs
https://api.github.com/users/Jeevesh8/received_events
https://api.github.com/users/Jeevesh8/repos
https://api.github.com/users/Jeevesh8/starred{/owner}{/repo}
https://api.github.com/users/Jeevesh8/subscriptions
User
https://api.github.com/users/Jeevesh8
public
MEMBER
Only datasets like wikipedia, wiki40b, wiki_dpr and natural questions are available already processed on the HF google storage. This is used to download directly the arrow file instead of building it from the original data files.
2021-03-25T13:49:28Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-806780702
https://api.github.com/repos/huggingface/datasets/issues/2092
MDEyOklzc3VlQ29tbWVudDgwNjc4MDcwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/806780702/reactions
2021-03-25T13:49:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/806780702
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq How can we make sure that the data we upload on HuggingFace hub is available in form of preprocessed arrow files ?
2021-03-26T14:41:01Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-808272742
https://api.github.com/repos/huggingface/datasets/issues/2092
MDEyOklzc3VlQ29tbWVudDgwODI3Mjc0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/808272742/reactions
2021-03-26T14:41:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/808272742
https://avatars.githubusercontent.com/u/48825663?v=4
https://api.github.com/users/Jeevesh8/events{/privacy}
https://api.github.com/users/Jeevesh8/followers
https://api.github.com/users/Jeevesh8/following{/other_user}
https://api.github.com/users/Jeevesh8/gists{/gist_id}
https://github.com/Jeevesh8
Jeevesh8
MDQ6VXNlcjQ4ODI1NjYz
https://api.github.com/users/Jeevesh8/orgs
https://api.github.com/users/Jeevesh8/received_events
https://api.github.com/users/Jeevesh8/repos
https://api.github.com/users/Jeevesh8/starred{/owner}{/repo}
https://api.github.com/users/Jeevesh8/subscriptions
User
https://api.github.com/users/Jeevesh8
public
MEMBER
We're still working on this :) This will be available soon
Users will be able to put their processed arrow files on the Hub
2021-03-26T18:37:08Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-808436031
https://api.github.com/repos/huggingface/datasets/issues/2092
MDEyOklzc3VlQ29tbWVudDgwODQzNjAzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/808436031/reactions
2021-03-26T18:37:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/808436031
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Hi! You can now use `Dataset.push_to_hub` to store preprocessed files on the Hub.

And to avoid downloading preprocessed files, you can use streaming by setting `streaming=True` in `load_dataset`.
2022-06-01T16:48:06Z
https://github.com/huggingface/datasets/issues/2092#issuecomment-1143866467
https://api.github.com/repos/huggingface/datasets/issues/2092
IC_kwDODunzps5ELgRj
https://api.github.com/repos/huggingface/datasets/issues/comments/1143866467/reactions
2022-06-01T16:49:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1143866467
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?