Flexible Loader
MEMBER
Ideally `save_to_disk` should save in a format compatible with load_dataset, wdyt ?
2025-03-13T11:14:58Z
https://github.com/huggingface/datasets/issues/7442#issuecomment-2720889843
https://api.github.com/repos/huggingface/datasets/issues/7442
IC_kwDODunzps6iLXvz
https://api.github.com/repos/huggingface/datasets/issues/comments/2720889843/reactions
2025-03-13T11:14:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2720889843
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Ideally `save_to_disk` should save in a format compatible with load_dataset, wdyt ?

That would be perfect if not at least a flexible loader.
2025-03-17T20:35:06Z
https://github.com/huggingface/datasets/issues/7442#issuecomment-2730821237
https://api.github.com/repos/huggingface/datasets/issues/7442
IC_kwDODunzps6ixQZ1
https://api.github.com/repos/huggingface/datasets/issues/comments/2730821237/reactions
2025-03-17T20:35:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2730821237
https://avatars.githubusercontent.com/u/13894030?v=4
https://api.github.com/users/dipta007/events{/privacy}
https://api.github.com/users/dipta007/followers
https://api.github.com/users/dipta007/following{/other_user}
https://api.github.com/users/dipta007/gists{/gist_id}
https://github.com/dipta007
dipta007
MDQ6VXNlcjEzODk0MDMw
https://api.github.com/users/dipta007/orgs
https://api.github.com/users/dipta007/received_events
https://api.github.com/users/dipta007/repos
https://api.github.com/users/dipta007/starred{/owner}{/repo}
https://api.github.com/users/dipta007/subscriptions
User
https://api.github.com/users/dipta007
public
NONE
@lhoestq For now, you can use this small utility library: [nanoml](https://pypi.org/project/nanoml/)
```python
from nanoml.data import load_dataset_flexible
```

I actively develop and maintain this utility library. Open to contributors. Please open issues, PR, or feature requests.
2025-03-27T23:58:01Z
https://github.com/huggingface/datasets/issues/7442#issuecomment-2759848087
https://api.github.com/repos/huggingface/datasets/issues/7442
IC_kwDODunzps6kf_CX
https://api.github.com/repos/huggingface/datasets/issues/comments/2759848087/reactions
2025-03-27T23:58:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2759848087
https://avatars.githubusercontent.com/u/13894030?v=4
https://api.github.com/users/dipta007/events{/privacy}
https://api.github.com/users/dipta007/followers
https://api.github.com/users/dipta007/following{/other_user}
https://api.github.com/users/dipta007/gists{/gist_id}
https://github.com/dipta007
dipta007
MDQ6VXNlcjEzODk0MDMw
https://api.github.com/users/dipta007/orgs
https://api.github.com/users/dipta007/received_events
https://api.github.com/users/dipta007/repos
https://api.github.com/users/dipta007/starred{/owner}{/repo}
https://api.github.com/users/dipta007/subscriptions
User
https://api.github.com/users/dipta007
public
### Feature request

Can we have a utility function that will use `load_from_disk` when given the local path and `load_dataset` if given an HF dataset?

It can be something as simple as this one:

```
def load_hf_dataset(path_or_name):
    if os.path.exists(path_or_name):
        return load_from_disk(path_or_name)
    else:
        return load_dataset(path_or_name)
```

### Motivation

This can be done inside the user codebase, too, but in my experience, it becomes repetitive code.

### Your contribution

I can open a pull request.