Add raw data files to the Hub with GitHub LFS for canonical dataset
MEMBER
Hi @zlucia, I would actually suggest hosting the dataset as a huggingface.co-hosted dataset.

The only difference with a "canonical"/legacy dataset is that it's nested under an organization (here `stanford` or `stanfordnlp` for instance – completely up to you) but then you can upload your data using git-lfs (unlike "canonical" datasets where we don't host the data)

Let me know if this fits your use case!

cc'ing @osanseviero @lhoestq and rest of the team 🤗
2021-10-26T08:26:36Z
https://github.com/huggingface/datasets/issues/3164#issuecomment-951704742
https://api.github.com/repos/huggingface/datasets/issues/3164
IC_kwDODunzps44udym
https://api.github.com/repos/huggingface/datasets/issues/comments/951704742/reactions
2021-10-26T08:26:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/951704742
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
MEMBER
Hi @zlucia,

As @julien-c pointed out, the way to store/host raw data files in our Hub is by using what we call "community" datasets:
- either at your personal namespace: `load_dataset("zlucia/casehold")`
- or at an organization namespace: for example, if you create the organization `reglab`, then `load_dataset("reglab/casehold")`

Please note that "canonical" datasets do not normally store/host their raw data at our Hub, but in a third-party server. For "canonical" datasets, we just host the "loading script", that is, a Python script that downloads the raw data from a third-party server, creates the HuggingFace dataset from it and caches it locally.

In order to create an organization namespace in our Hub, please follow this link: https://huggingface.co/organizations/new

There are already many organizations at our Hub (complete list here: https://huggingface.co/organizations), such as:
- Stanford CRFM: https://huggingface.co/stanford-crfm
- Stanford NLP: https://huggingface.co/stanfordnlp
- Stanford CS329S: Machine Learning Systems Design: https://huggingface.co/stanford-cs329s

Also note that you in your organization namespace:
- you can add any number of members
- you can store both raw datasets and models, and those can be immediately accessed using `datasets` and `transformers`

Once you have created an organization, these are the steps to upload/host a raw dataset: 
- The no-code procedure: https://huggingface.co/docs/datasets/upload_dataset.html
- Using the command line (terminal): https://huggingface.co/docs/datasets/share.html#add-a-community-dataset

Please, feel free to ping me if you have any further questions or need help.

2021-10-29T12:50:53Z
https://github.com/huggingface/datasets/issues/3164#issuecomment-954717863
https://api.github.com/repos/huggingface/datasets/issues/3164
IC_kwDODunzps4459an
https://api.github.com/repos/huggingface/datasets/issues/comments/954717863/reactions
2021-10-29T12:50:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/954717863
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Ah I see, I think I was unclear whether there were benefits to uploading a canonical dataset vs. a community provided dataset. Thanks for clarifying. I'll see if we want to create an organization namespace and otherwise, will upload the dataset under my personal namespace.
2021-10-30T19:54:51Z
https://github.com/huggingface/datasets/issues/3164#issuecomment-955584329
https://api.github.com/repos/huggingface/datasets/issues/3164
IC_kwDODunzps449Q9J
https://api.github.com/repos/huggingface/datasets/issues/comments/955584329/reactions
2021-10-30T19:54:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/955584329
https://avatars.githubusercontent.com/u/40370937?v=4
https://api.github.com/users/zlucia/events{/privacy}
https://api.github.com/users/zlucia/followers
https://api.github.com/users/zlucia/following{/other_user}
https://api.github.com/users/zlucia/gists{/gist_id}
https://github.com/zlucia
zlucia
MDQ6VXNlcjQwMzcwOTM3
https://api.github.com/users/zlucia/orgs
https://api.github.com/users/zlucia/received_events
https://api.github.com/users/zlucia/repos
https://api.github.com/users/zlucia/starred{/owner}{/repo}
https://api.github.com/users/zlucia/subscriptions
User
https://api.github.com/users/zlucia
public
I'm interested in sharing the CaseHOLD dataset (https://arxiv.org/abs/2104.08671) as a canonical dataset on the HuggingFace Hub and would like to add the raw data files to the Hub with GitHub LFS, since it seems like a more sustainable long term storage solution, compared to other storage solutions available to my team. From what I can tell, this option is not immediately supported if one follows the sharing steps detailed here: [https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset](https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset), since GitHub LFS is not supported for public forks. Is there a way to request this? Thanks!