Support streaming cfq dataset
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-06-27T17:23:14Z
https://github.com/huggingface/datasets/pull/4579#issuecomment-1167643389
https://api.github.com/repos/huggingface/datasets/issues/4579
IC_kwDODunzps5FmNL9
https://api.github.com/repos/huggingface/datasets/issues/comments/1167643389/reactions
2022-07-04T19:35:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1167643389
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
@lhoestq I've been refactoring a little the code:
- Use less RAM by loading only the required samples: only if its index is in the splits file
- Start yielding "earlier" in streaming mode: for each `split_idx`:
  - either yield from buffer
  - or iterate over samples and either yield or buffer the sample
  
  The speed gain obviously depends on how the indexes are sorted in the split file:
    - Best case: indices are [1, 2, 3]
    - Worst case (no speed gain): indices are [3, 1, 2] or [3, 2, 1]

Let me know what you think.
2022-06-30T16:30:17Z
https://github.com/huggingface/datasets/pull/4579#issuecomment-1171431628
https://api.github.com/repos/huggingface/datasets/issues/4579
IC_kwDODunzps5F0qDM
https://api.github.com/repos/huggingface/datasets/issues/comments/1171431628/reactions
2022-06-30T16:30:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1171431628
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I have to update the dummy data so that it aligns with the real data (inside the archive, the samples file `dataset.json` is the last member).
2022-06-30T16:42:21Z
https://github.com/huggingface/datasets/pull/4579#issuecomment-1171447717
https://api.github.com/repos/huggingface/datasets/issues/4579
IC_kwDODunzps5F0t-l
https://api.github.com/repos/huggingface/datasets/issues/comments/1171447717/reactions
2022-07-01T09:21:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1171447717
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
There is an issue when testing `test_load_dataset_cfq` with dummy data:
- `MockDownloadManager.iter_archive` yields FIRST `'cfq/dataset.json'`
- [`Streaming`]`DownloadManager.iter_archive` yields LAST `'cfq/dataset.json'` when using real data tar.gz archive

Note that this issue arises only with dummy data: loading the real dataset works smoothly for all configurations: I recreated the `dataset_infos.json` file to check it (it generated the same file).
2022-07-01T09:22:04Z
https://github.com/huggingface/datasets/pull/4579#issuecomment-1172135027
https://api.github.com/repos/huggingface/datasets/issues/4579
IC_kwDODunzps5F3Vxz
https://api.github.com/repos/huggingface/datasets/issues/comments/1172135027/reactions
2022-07-01T09:22:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1172135027
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
This PR should be merged first:
- #4611
2022-07-01T11:51:38Z
https://github.com/huggingface/datasets/pull/4579#issuecomment-1172263129
https://api.github.com/repos/huggingface/datasets/issues/4579
IC_kwDODunzps5F31DZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1172263129/reactions
2022-07-01T11:51:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1172263129
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Impressive, thank you ! :o 

feel free to merge master into this branch, now that the files order is respected. You can merge if the CI is green :)
2022-07-04T14:37:25Z
https://github.com/huggingface/datasets/pull/4579#issuecomment-1173893854
https://api.github.com/repos/huggingface/datasets/issues/4579
IC_kwDODunzps5F-DLe
https://api.github.com/repos/huggingface/datasets/issues/comments/1173893854/reactions
2022-07-04T14:37:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1173893854
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Support streaming cfq dataset.