Loading Data from HDF files
NONE
I'm currently working on bringing [Ecoset](https://www.pnas.org/doi/10.1073/pnas.2011417118) to huggingface datasets and I would second this request...
2022-05-31T12:41:41Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1142084085
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5EEtH1
https://api.github.com/repos/huggingface/datasets/issues/comments/1142084085/reactions
2022-05-31T12:41:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1142084085
https://avatars.githubusercontent.com/u/34550289?v=4
https://api.github.com/users/DiGyt/events{/privacy}
https://api.github.com/users/DiGyt/followers
https://api.github.com/users/DiGyt/following{/other_user}
https://api.github.com/users/DiGyt/gists{/gist_id}
https://github.com/DiGyt
DiGyt
MDQ6VXNlcjM0NTUwMjg5
https://api.github.com/users/DiGyt/orgs
https://api.github.com/users/DiGyt/received_events
https://api.github.com/users/DiGyt/repos
https://api.github.com/users/DiGyt/starred{/owner}{/repo}
https://api.github.com/users/DiGyt/subscriptions
User
https://api.github.com/users/DiGyt
public
NONE
I would also like this support or something similar. Geospatial datasets come in netcdf which is derived from hdf5, or zarr. I've gotten zarr stores to work with datasets and streaming, but it takes awhile to convert the data to zarr if it's not stored in that natively. 
2022-06-15T22:52:52Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1157025394
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5E9s5y
https://api.github.com/repos/huggingface/datasets/issues/comments/1157025394/reactions
2022-06-15T22:52:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1157025394
https://avatars.githubusercontent.com/u/7170359?v=4
https://api.github.com/users/jacobbieker/events{/privacy}
https://api.github.com/users/jacobbieker/followers
https://api.github.com/users/jacobbieker/following{/other_user}
https://api.github.com/users/jacobbieker/gists{/gist_id}
https://github.com/jacobbieker
jacobbieker
MDQ6VXNlcjcxNzAzNTk=
https://api.github.com/users/jacobbieker/orgs
https://api.github.com/users/jacobbieker/received_events
https://api.github.com/users/jacobbieker/repos
https://api.github.com/users/jacobbieker/starred{/owner}{/repo}
https://api.github.com/users/jacobbieker/subscriptions
User
https://api.github.com/users/jacobbieker
public
CONTRIBUTOR
@mariosasko , I would like to contribute on this "good second issue" . Is there anything in the works for this Issue or can I go ahead ? 

2022-07-31T20:37:09Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1200495670
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5Hjhw2
https://api.github.com/repos/huggingface/datasets/issues/comments/1200495670/reactions
2022-07-31T20:37:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1200495670
https://avatars.githubusercontent.com/u/20517962?v=4
https://api.github.com/users/VijayKalmath/events{/privacy}
https://api.github.com/users/VijayKalmath/followers
https://api.github.com/users/VijayKalmath/following{/other_user}
https://api.github.com/users/VijayKalmath/gists{/gist_id}
https://github.com/VijayKalmath
VijayKalmath
MDQ6VXNlcjIwNTE3OTYy
https://api.github.com/users/VijayKalmath/orgs
https://api.github.com/users/VijayKalmath/received_events
https://api.github.com/users/VijayKalmath/repos
https://api.github.com/users/VijayKalmath/starred{/owner}{/repo}
https://api.github.com/users/VijayKalmath/subscriptions
User
https://api.github.com/users/VijayKalmath
public
COLLABORATOR
Hi @VijayKalmath! As far as I know, nobody is working on it, so feel free to take over. Also, before you start, I suggest you comment `#self-assign` on this issue to assign it to yourself.
2022-08-08T14:53:52Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1208235255
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5IBDT3
https://api.github.com/repos/huggingface/datasets/issues/comments/1208235255/reactions
2022-08-08T14:53:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1208235255
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
#self-assign
2022-08-08T15:54:45Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1208305964
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5IBUks
https://api.github.com/repos/huggingface/datasets/issues/comments/1208305964/reactions
2022-08-08T15:54:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1208305964
https://avatars.githubusercontent.com/u/20517962?v=4
https://api.github.com/users/VijayKalmath/events{/privacy}
https://api.github.com/users/VijayKalmath/followers
https://api.github.com/users/VijayKalmath/following{/other_user}
https://api.github.com/users/VijayKalmath/gists{/gist_id}
https://github.com/VijayKalmath
VijayKalmath
MDQ6VXNlcjIwNTE3OTYy
https://api.github.com/users/VijayKalmath/orgs
https://api.github.com/users/VijayKalmath/received_events
https://api.github.com/users/VijayKalmath/repos
https://api.github.com/users/VijayKalmath/starred{/owner}{/repo}
https://api.github.com/users/VijayKalmath/subscriptions
User
https://api.github.com/users/VijayKalmath
public
NONE
Hey @mariosasko can you assign this issue to me !!
2023-10-09T06:57:54Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1752439342
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5odBou
https://api.github.com/repos/huggingface/datasets/issues/comments/1752439342/reactions
2023-10-09T06:57:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1752439342
https://avatars.githubusercontent.com/u/95092167?v=4
https://api.github.com/users/zutarich/events{/privacy}
https://api.github.com/users/zutarich/followers
https://api.github.com/users/zutarich/following{/other_user}
https://api.github.com/users/zutarich/gists{/gist_id}
https://github.com/zutarich
zutarich
U_kgDOBar9xw
https://api.github.com/users/zutarich/orgs
https://api.github.com/users/zutarich/received_events
https://api.github.com/users/zutarich/repos
https://api.github.com/users/zutarich/starred{/owner}{/repo}
https://api.github.com/users/zutarich/subscriptions
User
https://api.github.com/users/zutarich
public
NONE
So basically, we just need to load HDF5 files to Parquet?

e.g. Like this? https://stackoverflow.com/questions/46157709/converting-hdf5-to-parquet-without-loading-into-memory
2023-12-27T20:48:53Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-1870617882
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps5vf10a
https://api.github.com/repos/huggingface/datasets/issues/comments/1870617882/reactions
2023-12-27T20:49:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1870617882
https://avatars.githubusercontent.com/u/12627125?v=4
https://api.github.com/users/shermansiu/events{/privacy}
https://api.github.com/users/shermansiu/followers
https://api.github.com/users/shermansiu/following{/other_user}
https://api.github.com/users/shermansiu/gists{/gist_id}
https://github.com/shermansiu
shermansiu
MDQ6VXNlcjEyNjI3MTI1
https://api.github.com/users/shermansiu/orgs
https://api.github.com/users/shermansiu/received_events
https://api.github.com/users/shermansiu/repos
https://api.github.com/users/shermansiu/starred{/owner}{/repo}
https://api.github.com/users/shermansiu/subscriptions
User
https://api.github.com/users/shermansiu
public
CONTRIBUTOR
#self-assign
Hi! I'd love to take a shot at implementing this feature to allow loading HDF5 datasets via Hugging Face Datasets. Will keep you updated here!
2025-06-18T19:26:59Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-2985455454
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps6x8m9e
https://api.github.com/repos/huggingface/datasets/issues/comments/2985455454/reactions
2025-06-18T19:26:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2985455454
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
CONTRIBUTOR
Hey @mariosasko,
I've opened a PR to address this: #7625
It adds a new h5folder loader to support .h5 files via load_dataset("h5folder", data_dir=...).
Feedback welcome!
2025-06-19T05:41:22Z
https://github.com/huggingface/datasets/issues/3113#issuecomment-2986679639
https://api.github.com/repos/huggingface/datasets/issues/3113
IC_kwDODunzps6yBR1X
https://api.github.com/repos/huggingface/datasets/issues/comments/2986679639/reactions
2025-06-19T05:41:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2986679639
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
**Is your feature request related to a problem? Please describe.**
More often than not I come along big HDF datasets, and currently there is no straight forward way to feed them to a dataset.

**Describe the solution you'd like**
I would love to see a `from_h5` method that gets an interface implemented by the user on how items are extracted from dataset (in case of multiple datasets containing elements like arrays and metadata and etc).

**Describe alternatives you've considered**
Currently I manually load hdf files using `h5py` and implement PyTorch dataset interface. For small h5 files I load them into a pandas dataframe and use `from_pandas` function in the `datasets` package to load them, but for big datasets this is not feasible.

**Additional context**
HDF files are widespread throughout different domains and are one of the go to's for many researchers/scientists/engineers who work with numerical data. Given `datasets`' usecases have outgrown NLP use cases, it will make a lot of sense focusing on things like supporting HDF files.
