load `streaming=True` dataset with downloaded cache
MEMBER
When you pass `streaming=True`, the cache is ignored. The remote data URL is used instead and the data is streamed from the remote server.
2024-07-11T11:54:18Z
https://github.com/huggingface/datasets/issues/7040#issuecomment-2222732270
https://api.github.com/repos/huggingface/datasets/issues/7040
IC_kwDODunzps6EfDPu
https://api.github.com/repos/huggingface/datasets/issues/comments/2222732270/reactions
2024-07-11T11:54:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222732270
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thanks for your reply! So is there any solution to get my expected behavior besides clone the whole repo ? Or could I adjust my script to load the downloaded arrow files and generate the dataset streamingly?
2024-07-11T14:11:54Z
https://github.com/huggingface/datasets/issues/7040#issuecomment-2223046727
https://api.github.com/repos/huggingface/datasets/issues/7040
IC_kwDODunzps6EgQBH
https://api.github.com/repos/huggingface/datasets/issues/comments/2223046727/reactions
2024-07-11T14:11:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2223046727
https://avatars.githubusercontent.com/u/39429965?v=4
https://api.github.com/users/wanghaoyucn/events{/privacy}
https://api.github.com/users/wanghaoyucn/followers
https://api.github.com/users/wanghaoyucn/following{/other_user}
https://api.github.com/users/wanghaoyucn/gists{/gist_id}
https://github.com/wanghaoyucn
wanghaoyucn
MDQ6VXNlcjM5NDI5OTY1
https://api.github.com/users/wanghaoyucn/orgs
https://api.github.com/users/wanghaoyucn/received_events
https://api.github.com/users/wanghaoyucn/repos
https://api.github.com/users/wanghaoyucn/starred{/owner}{/repo}
https://api.github.com/users/wanghaoyucn/subscriptions
User
https://api.github.com/users/wanghaoyucn
public
### Describe the bug

We build a dataset which contains several hdf5 files and write a script using `h5py` to generate the dataset. The hdf5 files are large and the processed dataset cache takes more disk space. So we hope to try streaming iterable dataset. Unfortunately, `h5py` can't convert a remote URL into a hdf5 file descriptor. So we use `fsspec` as an interface like below:

```python
def _generate_examples(self, filepath, split):
        for file in filepath:
            with fsspec.open(file, "rb") as fs:
                with h5py.File(fs, "r") as fp:
                    # for event_id in sorted(list(fp.keys())):
                    event_ids = list(fp.keys())
                    ......
```

### Steps to reproduce the bug

The `fsspec` works, but it takes 10+ min to print the first 10 examples, which is even longer than the downloading time. I'm not sure if it just caches the whole hdf5 file and generates the examples.

### Expected behavior

So does the following make sense so far?

1. download the files
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True)
```

2. load the iterable dataset faster (using the raw file cache at path `.cache/huggingface/datasets/downloads`)
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True, streaming=true)
```

I made some tests, but the code above can't get the expected result. I'm not sure if this is supported. I also find the issue #6327 . It seemed similar to mine, but I couldn't find a solution.

### Environment info

- `datasets` = 2.18.0
- `h5py` = 3.10.0
- `fsspec` = 2023.10.0