Dataset sizes are in MiB instead of MB in dataset cards
MEMBER
Example of bulk edit: https://huggingface.co/datasets/aeslc/discussions/5
2023-04-05T08:16:39Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497099476
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZO-zU
https://api.github.com/repos/huggingface/datasets/issues/comments/1497099476/reactions
2023-04-05T08:16:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497099476
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`
2023-04-05T08:30:30Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497116770
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZPDBi
https://api.github.com/repos/huggingface/datasets/issues/comments/1497116770/reactions
2023-04-05T08:30:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497116770
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
MEMBER
I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.
2023-04-05T08:43:03Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497131619
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZPGpj
https://api.github.com/repos/huggingface/datasets/issues/comments/1497131619/reactions
2023-04-05T08:49:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497131619
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
yes, worst case, we have this in structured data:

<img width="337" alt="image" src="https://user-images.githubusercontent.com/326577/230037051-06caddcb-08c8-4953-a710-f3d122917db3.png">

2023-04-05T09:14:04Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497171504
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZPQYw
https://api.github.com/repos/huggingface/datasets/issues/comments/1497171504/reactions
2023-04-05T09:14:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497171504
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
MEMBER
I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files
2023-04-05T09:39:50Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497203478
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZPYMW
https://api.github.com/repos/huggingface/datasets/issues/comments/1497203478/reactions
2023-04-05T09:59:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497203478
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
Nice. Is it another loop? Because in https://huggingface.co/datasets/amazon_us_reviews/discussions/2/files we have `32377.29 MB` for example
2023-04-05T09:54:08Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497221382
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZPckG
https://api.github.com/repos/huggingface/datasets/issues/comments/1497221382/reactions
2023-04-05T09:54:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497221382
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.
2023-04-05T10:00:15Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497229249
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZPefB
https://api.github.com/repos/huggingface/datasets/issues/comments/1497229249/reactions
2023-04-05T10:03:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497229249
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
The bulk edit parsed 751 canonical datasets and updated 166.
2023-04-05T14:09:04Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497555826
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZQuNy
https://api.github.com/repos/huggingface/datasets/issues/comments/1497555826/reactions
2023-04-05T14:09:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497555826
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">

2023-04-05T14:09:41Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1497556903
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZQuen
https://api.github.com/repos/huggingface/datasets/issues/comments/1497556903/reactions
2023-04-05T14:10:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1497556903
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [x] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [x] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6
2023-04-06T14:54:02Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1499202694
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5ZXASG
https://api.github.com/repos/huggingface/datasets/issues/comments/1499202694/reactions
2023-12-21T10:19:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1499202694
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
should we force merge the PR and close this issue?
2023-09-25T12:06:57Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1733550943
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5nU-Nf
https://api.github.com/repos/huggingface/datasets/issues/comments/1733550943/reactions
2023-09-25T12:06:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1733550943
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
I merged the PRs for "scicite" and "scifact".
2023-12-21T10:20:27Z
https://github.com/huggingface/datasets/issues/5708#issuecomment-1866003212
https://api.github.com/repos/huggingface/datasets/issues/5708
IC_kwDODunzps5vOPMM
https://api.github.com/repos/huggingface/datasets/issues/comments/1866003212/reactions
2023-12-21T10:20:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1866003212
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations