Add MedImg for streaming
NONE
@mariosasko, @lhoestq, @albertvillanova
Hello! Can anyone help? or can you guys suggest who can help with this?
2024-05-31T03:15:45Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2141160791
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps5_n4VX
https://api.github.com/repos/huggingface/datasets/issues/comments/2141160791/reactions
2024-05-31T03:15:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2141160791
https://avatars.githubusercontent.com/u/72926928?v=4
https://api.github.com/users/lhallee/events{/privacy}
https://api.github.com/users/lhallee/followers
https://api.github.com/users/lhallee/following{/other_user}
https://api.github.com/users/lhallee/gists{/gist_id}
https://github.com/lhallee
lhallee
MDQ6VXNlcjcyOTI2OTI4
https://api.github.com/users/lhallee/orgs
https://api.github.com/users/lhallee/received_events
https://api.github.com/users/lhallee/repos
https://api.github.com/users/lhallee/starred{/owner}{/repo}
https://api.github.com/users/lhallee/subscriptions
User
https://api.github.com/users/lhallee
public
MEMBER
Hi ! Feel free to download the dataset and create a `Dataset` object with it.

Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)
2024-05-31T10:26:34Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2141722641
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps5_qBgR
https://api.github.com/repos/huggingface/datasets/issues/comments/2141722641/reactions
2024-05-31T10:26:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2141722641
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Hi ! Feel free to download the dataset and create a `Dataset` object with it.
> 
> Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

The dataset is several TB in total, which I do not have the resources to handle.
2024-06-03T14:40:09Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2145378687
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps5_3-F_
https://api.github.com/repos/huggingface/datasets/issues/comments/2145378687/reactions
2024-06-03T14:40:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2145378687
https://avatars.githubusercontent.com/u/72926928?v=4
https://api.github.com/users/lhallee/events{/privacy}
https://api.github.com/users/lhallee/followers
https://api.github.com/users/lhallee/following{/other_user}
https://api.github.com/users/lhallee/gists{/gist_id}
https://github.com/lhallee
lhallee
MDQ6VXNlcjcyOTI2OTI4
https://api.github.com/users/lhallee/orgs
https://api.github.com/users/lhallee/received_events
https://api.github.com/users/lhallee/repos
https://api.github.com/users/lhallee/starred{/owner}{/repo}
https://api.github.com/users/lhallee/subscriptions
User
https://api.github.com/users/lhallee
public
NONE
Hi @lhoestq and @albertvillanova , just following up about this.
2024-09-05T13:58:03Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2331758337
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps6K-88B
https://api.github.com/repos/huggingface/datasets/issues/comments/2331758337/reactions
2024-09-05T13:58:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2331758337
https://avatars.githubusercontent.com/u/72926928?v=4
https://api.github.com/users/lhallee/events{/privacy}
https://api.github.com/users/lhallee/followers
https://api.github.com/users/lhallee/following{/other_user}
https://api.github.com/users/lhallee/gists{/gist_id}
https://github.com/lhallee
lhallee
MDQ6VXNlcjcyOTI2OTI4
https://api.github.com/users/lhallee/orgs
https://api.github.com/users/lhallee/received_events
https://api.github.com/users/lhallee/repos
https://api.github.com/users/lhallee/starred{/owner}{/repo}
https://api.github.com/users/lhallee/subscriptions
User
https://api.github.com/users/lhallee
public
MEMBER
for big datasets you can push_to_hub one part at a time (e.g. as different splits) and merge the parts (just a simple modification in the YAML part of the README)
2024-09-05T15:49:14Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2332088236
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps6LANes
https://api.github.com/repos/huggingface/datasets/issues/comments/2332088236/reactions
2024-09-05T15:49:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2332088236
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Sure, that makes sense. However, isn't there a size limit to what typical users can push?
2024-09-05T16:19:53Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2332153459
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps6LAdZz
https://api.github.com/repos/huggingface/datasets/issues/comments/2332153459/reactions
2024-09-05T16:19:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2332153459
https://avatars.githubusercontent.com/u/72926928?v=4
https://api.github.com/users/lhallee/events{/privacy}
https://api.github.com/users/lhallee/followers
https://api.github.com/users/lhallee/following{/other_user}
https://api.github.com/users/lhallee/gists{/gist_id}
https://github.com/lhallee
lhallee
MDQ6VXNlcjcyOTI2OTI4
https://api.github.com/users/lhallee/orgs
https://api.github.com/users/lhallee/received_events
https://api.github.com/users/lhallee/repos
https://api.github.com/users/lhallee/starred{/owner}{/repo}
https://api.github.com/users/lhallee/subscriptions
User
https://api.github.com/users/lhallee
public
MEMBER
Yes there is a limit, simply let us know by email at datasets [at] huggingface.co - this way we can give you a storage grant also help making sure the dataset is all good for people to use it easily
2024-09-05T16:46:26Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2332202070
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps6LApRW
https://api.github.com/repos/huggingface/datasets/issues/comments/2332202070/reactions
2024-09-05T16:47:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2332202070
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Yes there is a limit, simply let us know by email at datasets [at] huggingface.co - this way we can give you a storage grant also help making sure the dataset is all good for people to use it easily

Got it, that would be great.
2024-09-05T16:53:53Z
https://github.com/huggingface/datasets/issues/6912#issuecomment-2332215466
https://api.github.com/repos/huggingface/datasets/issues/6912
IC_kwDODunzps6LAsiq
https://api.github.com/repos/huggingface/datasets/issues/comments/2332215466/reactions
2024-09-05T16:53:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2332215466
https://avatars.githubusercontent.com/u/72926928?v=4
https://api.github.com/users/lhallee/events{/privacy}
https://api.github.com/users/lhallee/followers
https://api.github.com/users/lhallee/following{/other_user}
https://api.github.com/users/lhallee/gists{/gist_id}
https://github.com/lhallee
lhallee
MDQ6VXNlcjcyOTI2OTI4
https://api.github.com/users/lhallee/orgs
https://api.github.com/users/lhallee/received_events
https://api.github.com/users/lhallee/repos
https://api.github.com/users/lhallee/starred{/owner}{/repo}
https://api.github.com/users/lhallee/subscriptions
User
https://api.github.com/users/lhallee
public
### Feature request

Host the MedImg dataset (similar to Imagenet but for biomedical images). 

### Motivation

There is a clear need for biomedical image foundation models and large scale biomedical datasets that are easily streamable. This would be an excellent tool for the biomedical community.

### Your contribution

MedImg can be found [here](https://www.cuilab.cn/medimg/#).