Add RVL-CDIP dataset
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-03-29T06:10:04Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1081441946
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5AdX6a
https://api.github.com/repos/huggingface/datasets/issues/comments/1081441946/reactions
2022-04-21T17:21:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1081441946
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
Thanks a lot for inputs. I'll use the URL suggested and check.

>  we need to implement the streamable (can't use os.path.join) and the non-streamable versions of _generate_examples.

Sure. I will check the reference and try this out, will get back to you if I face any issues.

> The labels-only data file URL doesn't work for me, so feel free to ask the authors whether they are OK with us hosting the file on the Hub/S3 (to speed up the streamable version)

Just checked. The author (Adam Harley) has responded positively and allowed us to host the file. Do I share the file with you for hosting it on Hub/S3 ?
2022-03-29T18:19:32Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1082224842
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5AgXDK
https://api.github.com/repos/huggingface/datasets/issues/comments/1082224842/reactions
2022-03-29T18:19:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1082224842
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
COLLABORATOR
> Just checked. The author (Adam Harley) has responded positively and allowed us to host the file. Do I share the file with you for hosting it on Hub/S3 ?

Yes, feel free to e-mail me the file. Then I'll create a repo under my namespace and push the file there. We run a GH action on a GH dataset after merging to create its repo on the Hub, so after this PR is merged, I'll push the file to the "official" namespace and update the download link.
2022-03-30T12:00:51Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1083051858
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5Ajg9S
https://api.github.com/repos/huggingface/datasets/issues/comments/1083051858/reactions
2022-03-30T12:00:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1083051858
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> You can use this URL to avoid manual download: https://drive.google.com/uc?export=download&id=0Bz1dfcnrpXM-MUt4cHNzUEFXcmc

For some reason, the direct download doesn't seem to work for me even with this URL. 
```
Downloading and preparing dataset rvl_cdip/default to ~/.cache/huggingface/datasets/rvl_cdip/default/1.0.0/ea152149e06310d60a9ef3c3020199dd4780bb952a773ba5aac6b57d59f12628...
Downloading data files: 100%|█████████████████████████████████████████████████████| 1/1 [00:00<00:00, 6307.22it/s]
{'rvl-cdip': '~/.cache/huggingface/datasets/downloads/07ef956a33750078d570d76fefe9fed49f7dc32ecf6e872d690de11e66bbe869'}
```
And this directory does not exist. Am I doing something wrong ?
To verify, I tried using [gdown](https://github.com/wkentaro/gdown) for the above URL, we get the following : 
```
Access denied with the following error:

        Cannot retrieve the public link of the file. You may need to change
        the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:
```
----

> Yes, feel free to e-mail me the file. Then I'll create a repo under my namespace and push the file there. We run a GH action on a GH dataset after merging to create its repo on the Hub, so after this PR is merged, I'll push the file to the "official" namespace and update the download link.

Got it. I've sent you an email with the file. Thank you.
2022-03-30T15:39:31Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1083303557
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5AkeaF
https://api.github.com/repos/huggingface/datasets/issues/comments/1083303557/reactions
2022-03-30T17:01:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1083303557
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
Actually this URL works for direct download :
`https://drive.google.com/uc?export=download&confirm=pbef&id=0Bz1dfcnrpXM-MUt4cHNzUEFXcmc`
Ref : https://github.com/wkentaro/gdown/issues/146#issuecomment-1042382215

I'm working on the streamable versions of _generate_examples as well, will update you regarding this.
2022-04-04T14:28:55Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1087630677
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5A0-1V
https://api.github.com/repos/huggingface/datasets/issues/comments/1087630677/reactions
2022-04-04T14:30:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1087630677
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
COLLABORATOR
Google Drive is a tricky host, and it's easy to exceed daily download quota limits, so if we are allowed to host the `rvl-cdip.tar.gz` file, I can push it to the Hub.
2022-04-04T14:53:48Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1087658961
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5A1FvR
https://api.github.com/repos/huggingface/datasets/issues/comments/1087658961/reactions
2022-04-04T14:53:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1087658961
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Just checked, the authors have agreed. He mentioned that he had complaints about the GDrive link.
You can push it to the Hub and share the link. :)
2022-04-04T15:47:18Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1087722251
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5A1VML
https://api.github.com/repos/huggingface/datasets/issues/comments/1087722251/reactions
2022-04-04T15:47:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1087722251
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
I have added :
- streaming support for rvl-cdip.tar.gz file. [ Need to test this ]

Is it possible for you to upload the train.txt, test.txt, val.txt files separately to the Hub instead of labels_only.tar.gz file.
Currently during the tests in stream mode, we get : 
`NotImplementedError: Extraction protocol for TAR archives like 'https://huggingface.co/datasets/mariosasko/rvl_cdip/resolve/main/labels_only.tar.gz' is not implemented in streaming mode. Please use dl_manager.iter_archive instead.`
If the label files are present as .txt files then we can directly use dl_manager.download.



2022-04-05T15:37:12Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1088882095
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5A5wWv
https://api.github.com/repos/huggingface/datasets/issues/comments/1088882095/reactions
2022-04-05T15:37:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1088882095
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
COLLABORATOR
The rvl-cdip.tar.gz archive and txt files with the labels are on the Hub!
2022-04-08T11:21:11Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1092758281
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5BIisJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1092758281/reactions
2022-04-08T11:21:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1092758281
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
- Added 🤗 Hub download links.
- streamable and non-streamable versions of _generate_examples.
- Updated dummy data, both real and dummy dataset tests have passed.


2022-04-10T15:38:29Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1094298958
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5BOa1O
https://api.github.com/repos/huggingface/datasets/issues/comments/1094298958/reactions
2022-04-10T15:38:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1094298958
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
I've removed the extraction of the archive file locally as suggested. Let me know if any other changes are required. :)
2022-04-14T06:44:57Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1098765370
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5BfdQ6
https://api.github.com/repos/huggingface/datasets/issues/comments/1098765370/reactions
2022-04-14T06:44:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1098765370
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
The check for **Update Hub repositories / update-hub-repositories** has failed.

> https://github.com/huggingface/datasets/runs/6116502392?check_suite_focus=true


2022-04-22T06:21:17Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1106052050
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5B7QPS
https://api.github.com/repos/huggingface/datasets/issues/comments/1106052050/reactions
2022-04-22T06:21:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106052050
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
MEMBER
Hi ! Thanks for reporting ;) yes this CI job has been failing for a few days. I'm working on fixing it, and I'm manually running it on my side in the meantime
2022-04-22T09:02:25Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1106215018
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5B74Bq
https://api.github.com/repos/huggingface/datasets/issues/comments/1106215018/reactions
2022-04-22T09:02:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106215018
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Great. :D Thank you @lhoestq 
2022-04-22T09:55:07Z
https://github.com/huggingface/datasets/pull/4050#issuecomment-1106297709
https://api.github.com/repos/huggingface/datasets/issues/4050
IC_kwDODunzps5B8MNt
https://api.github.com/repos/huggingface/datasets/issues/comments/1106297709/reactions
2022-04-22T09:55:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106297709
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
Resolves #2762 

Dataset Request : Add RVL-CDIP dataset [#2762](https://github.com/huggingface/datasets/issues/2762)
This PR adds the RVL-CDIP dataset.

The dataset contains Google Drive link for download and wasn't getting downloaded automatically, so I have provided manual_download_instructions.
- I have added the dummy_data.zip as well.

Needed inputs on how I can run the real data and the dummy data tests for datasets with manual download ?

Inputs and suggestions for improvement are welcome. Thank you.