REUSE_DATASET_IF_EXISTS don't work 
COLLABORATOR
Hi! Unfortunately, the current behavior is to delete the downloaded data when this error happens. So, I've opened a PR that removes the problematic import to avoid losing data due to `apache_beam` not being installed (we host the preprocessed version of `natual_questions` on the HF GCS, so requiring `apache_beam` in that case doesn't make sense)
2023-09-27T14:07:40Z
https://github.com/huggingface/datasets/issues/6260#issuecomment-1737474847
https://api.github.com/repos/huggingface/datasets/issues/6260
IC_kwDODunzps5nj8Mf
https://api.github.com/repos/huggingface/datasets/issues/comments/1737474847/reactions
2023-09-27T14:07:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1737474847
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thanks for your reply. I met another question that I set `export HF_DATASETS_CACHE=/data/lxy/.cache` , but each time I run load_datasets, the datasets module still looking for NQ in the wrong default cache dir '/home/lxy/.cache' 。How to avoid this incorrect behavior. I am sure HF_DATASETS_CACHE was set correctly since I use echo & to check it.
![image](https://github.com/huggingface/datasets/assets/88258534/e7029f27-b9f9-496c-8948-6234ef695646)
by the way I delete the file in '/home/lxy/.cache' since I found there has some kb size file seems useless.
2023-09-28T05:50:45Z
https://github.com/huggingface/datasets/issues/6260#issuecomment-1738514175
https://api.github.com/repos/huggingface/datasets/issues/6260
IC_kwDODunzps5nn57_
https://api.github.com/repos/huggingface/datasets/issues/comments/1738514175/reactions
2023-09-28T05:51:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1738514175
https://avatars.githubusercontent.com/u/88258534?v=4
https://api.github.com/users/rangehow/events{/privacy}
https://api.github.com/users/rangehow/followers
https://api.github.com/users/rangehow/following{/other_user}
https://api.github.com/users/rangehow/gists{/gist_id}
https://github.com/rangehow
rangehow
MDQ6VXNlcjg4MjU4NTM0
https://api.github.com/users/rangehow/orgs
https://api.github.com/users/rangehow/received_events
https://api.github.com/users/rangehow/repos
https://api.github.com/users/rangehow/starred{/owner}{/repo}
https://api.github.com/users/rangehow/subscriptions
User
https://api.github.com/users/rangehow
public
COLLABORATOR
You need to set this variable before the `datasets` import. Then, you can use `import datasets; datasets.config.HF_DATASETS_CACHE` to verify the cache location.
2023-09-28T13:57:31Z
https://github.com/huggingface/datasets/issues/6260#issuecomment-1739269280
https://api.github.com/repos/huggingface/datasets/issues/6260
IC_kwDODunzps5nqySg
https://api.github.com/repos/huggingface/datasets/issues/comments/1739269280/reactions
2023-09-28T13:57:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1739269280
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

I use the following code  to download natural_question dataset. Even though I have completely download it, the next time I run this code, the new download procedure will start and cover the original /data/lxy/NQ

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

---
Since I don't have apache_beam installed, it throw a exception. After I pip install apache_beam ,the download restart..

![image](https://github.com/huggingface/datasets/assets/88258534/f28ce7fe-29ea-4348-b87f-e69182a8bd41)


### Steps to reproduce the bug

run this two line code

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

### Expected behavior

Download behavior can be correctly follow DownloadMode

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.3