load_dataset in v2.11.0 raises "ValueError: seek of closed file" in np.load()
### Describe the bug

Hi,

I have some `dataset_load()` code of a custom offline dataset that works with datasets v2.10.1. 

```python
        ds = datasets.load_dataset(path=dataset_dir,
                                   name=configuration,
                                   data_dir=dataset_dir,
                                   cache_dir=cache_dir,
                                   aux_dir=aux_dir,
                                #    download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                   num_proc=18)
```

When upgrading datasets to 2.11.0, it fails with error

```
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/site-packages/datasets/load.py", line 1791, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/site-packages/datasets/builder.py", line 891, in download_and_prepare
    self._download_and_prepare(
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/site-packages/datasets/builder.py", line 1651, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/site-packages/datasets/builder.py", line 964, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ramon.casero/.cache/huggingface/modules/datasets_modules/datasets/71f67f69e6e00e139903a121f96b71f39b65a6b6aaeb0862e6a5da3a3f565b4c/mydataset.py", line 682, in _split_generators
    self.some_function()
  File "/home/ramon.casero/.cache/huggingface/modules/datasets_modules/datasets/71f67f69e6e00e139903a121f96b71f39b65a6b6aaeb0862e6a5da3a3f565b4c/mydataset.py", line 1314, in some_function()
    x_df = pd.DataFrame({'cell_type_descriptor': fp['x'].tolist()})
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/site-packages/numpy/lib/npyio.py", line 248, in __getitem__
    bytes = self.zip.open(key)
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/zipfile.py", line 1530, in open
    fheader = zef_file.read(sizeFileHeader)
  File "/home/ramon.casero/opt/miniconda3/envs/myenv/lib/python3.10/zipfile.py", line 744, in read
    self._file.seek(self._pos)
ValueError: seek of closed file
```


### Steps to reproduce the bug

Sorry, I cannot share the data or code because they are not mine to share, but the point of failure is a call in `some_function()`

```python
        with np.load(filename) as fp:
            x_df = pd.DataFrame({'feature': fp['x'].tolist()})
```

I'll try to generate a short snippet that reproduces the error.

### Expected behavior

I would expect that `load_dataset` works on the custom datasets generation script for v2.11.0 the same way it works for 2.10.1, without making `np.load()` give a `ValueError: seek of closed file` error.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- Huggingface_hub version: 0.12.0
- PyArrow version: 11.0.0
- Pandas version: 1.5.2
- numpy: 1.24.2
- This is an offline dataset that uses `datasets.config.HF_DATASETS_OFFLINE = True` in the generation script.
