Replace data URL in big_patent dataset and support streaming
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-27T10:12:36Z
https://github.com/huggingface/datasets/pull/4236#issuecomment-1110822730
https://api.github.com/repos/huggingface/datasets/issues/4236
IC_kwDODunzps5CNc9K
https://api.github.com/repos/huggingface/datasets/issues/comments/1110822730/reactions
2022-05-02T18:28:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1110822730
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
I first uploaded the data files to the Hub: I think it is a good option because we have git lfs to track versions and changes. Moreover people will be able to make PRs to propose updates on the data files.
- I would have preferred to upload it it to the "data" org namespace, but it is already taken (although not used): might be possible to take it?

As an alternative (and to be consistent with previous datasets), I also uploaded the data files to our AWS bucket.

We should decide which to use (now and for future datasets) and set it here before merging. We should remove the data files for the non-chosen option.

CC: @lhoestq @mariosasko @polinaeterna 
2022-04-28T17:19:14Z
https://github.com/huggingface/datasets/pull/4236#issuecomment-1112462835
https://api.github.com/repos/huggingface/datasets/issues/4236
IC_kwDODunzps5CTtXz
https://api.github.com/repos/huggingface/datasets/issues/comments/1112462835/reactions
2022-04-28T17:19:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1112462835
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Would it make sense to make the dataset a community one (so, create an organization for it) and store the script and the data in a single repository? Just as it is for most of the datasets. That way we can also access the data using a relative path inside the repo (that's not the point though). The point is that to me it seems a bit more straightforward to store everything in one place. 

I guess the strong argument against this logic is that in this case the canonical version won't work... But maybe there is some redirecting mechanism I don't know about? :)

Anyway, I'm in favor of hosting data on the Hub instead of AWS :) 
2022-04-29T16:58:34Z
https://github.com/huggingface/datasets/pull/4236#issuecomment-1113528524
https://api.github.com/repos/huggingface/datasets/issues/4236
IC_kwDODunzps5CXxjM
https://api.github.com/repos/huggingface/datasets/issues/comments/1113528524/reactions
2022-04-29T16:58:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1113528524
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
COLLABORATOR
I also think storing everything in one place/single repository is the best option.

@polinaeterna Canonical datasets also support data files (see the [`red_caps` repo](https://huggingface.co/datasets/red_caps/tree/main) for instance) 
2022-05-02T11:36:20Z
https://github.com/huggingface/datasets/pull/4236#issuecomment-1114751216
https://api.github.com/repos/huggingface/datasets/issues/4236
IC_kwDODunzps5CccDw
https://api.github.com/repos/huggingface/datasets/issues/comments/1114751216/reactions
2022-05-02T11:36:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1114751216
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Thanks @polinaeterna and @mariosasko for your comments.

Yes, definitely it is much better to have everything in the same repo. 

I'm transferring their data files to the Hub under "big_patent" and deleting them from the other repo and AWS.
2022-05-02T16:02:02Z
https://github.com/huggingface/datasets/pull/4236#issuecomment-1115067932
https://api.github.com/repos/huggingface/datasets/issues/4236
IC_kwDODunzps5CdpYc
https://api.github.com/repos/huggingface/datasets/issues/comments/1115067932/reactions
2022-05-02T16:02:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1115067932
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
This PR replaces the Google Drive URL with our Hub one, once the data owners have approved to host their data on the Hub.

Moreover, this PR makes the dataset streamable.

Fix #4217.