How to load data from lakefs
COLLABORATOR
You can create a `pandas` DataFrame following [this](https://lakefs.io/data-version-control/dvc-using-python/) tutorial, and then convert this DataFrame to a `Dataset` with `datasets.Dataset.from_pandas`. For larger datasets (to memory map them), you can use `Dataset.from_generator` with a generator function that reads lakeFS files with `s3fs`.
2023-12-06T15:55:46Z
https://github.com/huggingface/datasets/issues/6478#issuecomment-1843187665
https://api.github.com/repos/huggingface/datasets/issues/6478
IC_kwDODunzps5t3M_R
https://api.github.com/repos/huggingface/datasets/issues/comments/1843187665/reactions
2023-12-06T15:55:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1843187665
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@mariosasko  hello,
This can achieve and https://huggingface.co/datasets Does the same effect apply to the dataset? For example, downloading while using
2023-12-07T02:19:43Z
https://github.com/huggingface/datasets/issues/6478#issuecomment-1844096346
https://api.github.com/repos/huggingface/datasets/issues/6478
IC_kwDODunzps5t6q1a
https://api.github.com/repos/huggingface/datasets/issues/comments/1844096346/reactions
2023-12-07T02:19:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1844096346
https://avatars.githubusercontent.com/u/12895488?v=4
https://api.github.com/users/d710055071/events{/privacy}
https://api.github.com/users/d710055071/followers
https://api.github.com/users/d710055071/following{/other_user}
https://api.github.com/users/d710055071/gists{/gist_id}
https://github.com/d710055071
d710055071
MDQ6VXNlcjEyODk1NDg4
https://api.github.com/users/d710055071/orgs
https://api.github.com/users/d710055071/received_events
https://api.github.com/users/d710055071/repos
https://api.github.com/users/d710055071/starred{/owner}{/repo}
https://api.github.com/users/d710055071/subscriptions
User
https://api.github.com/users/d710055071
public
MEMBER
There is a blogspot from lakes on this topic: https://lakefs.io/blog/data-version-control-hugging-face-datasets/
2024-07-03T19:13:56Z
https://github.com/huggingface/datasets/issues/6478#issuecomment-2207016987
https://api.github.com/repos/huggingface/datasets/issues/6478
IC_kwDODunzps6DjGgb
https://api.github.com/repos/huggingface/datasets/issues/comments/2207016987/reactions
2024-07-03T19:13:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2207016987
https://avatars.githubusercontent.com/u/9143109?v=4
https://api.github.com/users/andimarafioti/events{/privacy}
https://api.github.com/users/andimarafioti/followers
https://api.github.com/users/andimarafioti/following{/other_user}
https://api.github.com/users/andimarafioti/gists{/gist_id}
https://github.com/andimarafioti
andimarafioti
MDQ6VXNlcjkxNDMxMDk=
https://api.github.com/users/andimarafioti/orgs
https://api.github.com/users/andimarafioti/received_events
https://api.github.com/users/andimarafioti/repos
https://api.github.com/users/andimarafioti/starred{/owner}{/repo}
https://api.github.com/users/andimarafioti/subscriptions
User
https://api.github.com/users/andimarafioti
public
My dataset is stored on the company's lakefs server. How can I write code to load the dataset? It would be great if I could provide code examples or provide some references
