FORCE_REDOWNLOAD raises "Directory not empty" exception on second run
MEMBER
Hi ! I couldn't reproduce your issue :/

It seems that `shutil.rmtree` failed. It is supposed to work even if the directory is not empty, but you still end up with `OSError: [Errno 39] Directory not empty:`. Can you make sure another process is not using this directory at the same time ?
2023-04-21T15:27:38Z
https://github.com/huggingface/datasets/issues/5736#issuecomment-1518000273
https://api.github.com/repos/huggingface/datasets/issues/5736
IC_kwDODunzps5aetiR
https://api.github.com/repos/huggingface/datasets/issues/comments/1518000273/reactions
2023-04-21T15:27:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1518000273
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I have the same error with `datasets==2.14.5` and `pyarrow==13.0.0`. Python 3.10.13
2023-10-12T16:19:16Z
https://github.com/huggingface/datasets/issues/5736#issuecomment-1759946479
https://api.github.com/repos/huggingface/datasets/issues/5736
IC_kwDODunzps5o5qbv
https://api.github.com/repos/huggingface/datasets/issues/comments/1759946479/reactions
2023-10-12T16:19:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1759946479
https://avatars.githubusercontent.com/u/2575047?v=4
https://api.github.com/users/WojciechKusa/events{/privacy}
https://api.github.com/users/WojciechKusa/followers
https://api.github.com/users/WojciechKusa/following{/other_user}
https://api.github.com/users/WojciechKusa/gists{/gist_id}
https://github.com/WojciechKusa
WojciechKusa
MDQ6VXNlcjI1NzUwNDc=
https://api.github.com/users/WojciechKusa/orgs
https://api.github.com/users/WojciechKusa/received_events
https://api.github.com/users/WojciechKusa/repos
https://api.github.com/users/WojciechKusa/starred{/owner}{/repo}
https://api.github.com/users/WojciechKusa/subscriptions
User
https://api.github.com/users/WojciechKusa
public
NONE
I have same error. Any workaround?
2023-11-30T07:16:57Z
https://github.com/huggingface/datasets/issues/5736#issuecomment-1833228332
https://api.github.com/repos/huggingface/datasets/issues/5736
IC_kwDODunzps5tRNgs
https://api.github.com/repos/huggingface/datasets/issues/comments/1833228332/reactions
2023-11-30T07:16:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1833228332
https://avatars.githubusercontent.com/u/5590702?v=4
https://api.github.com/users/duyvuleo/events{/privacy}
https://api.github.com/users/duyvuleo/followers
https://api.github.com/users/duyvuleo/following{/other_user}
https://api.github.com/users/duyvuleo/gists{/gist_id}
https://github.com/duyvuleo
duyvuleo
MDQ6VXNlcjU1OTA3MDI=
https://api.github.com/users/duyvuleo/orgs
https://api.github.com/users/duyvuleo/received_events
https://api.github.com/users/duyvuleo/repos
https://api.github.com/users/duyvuleo/starred{/owner}{/repo}
https://api.github.com/users/duyvuleo/subscriptions
User
https://api.github.com/users/duyvuleo
public
### Describe the bug

Running `load_dataset(..., download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` twice raises a `Directory not empty` exception on the second run.

### Steps to reproduce the bug

I cannot test this on datasets v2.11.0 due to #5711, but this happens in v2.10.1.

1. Set up a script `my_dataset.py` to generate and load an offline dataset.
2. Load it with 
    ```python
            ds = datasets.load_dataset(path=/path/to/my_dataset.py,
                                       name='toy',
                                       data_dir=/path/to/my_dataset.py,
                                       cache_dir=cache_dir,
                                       download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                       )
    ```
    It loads fine
    ```
    Dataset my_dataset downloaded and prepared to /path/to/cache/toy-..e05e/1.0.0/...5b4c. Subsequent calls will reuse this data.
    ```
3. Try to load it again with the same snippet and the splits are generated, but at the end of the loading process it raises the error
    ```
    2023-04-11 12:10:19,965: DEBUG:	open file: /path/to/cache/toy-..e05e/1.0.0/...5b4c.incomplete/dataset_info.json
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
        builder_instance.download_and_prepare(
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 852, in download_and_prepare
        with incomplete_dir(self._output_dir) as tmp_output_dir:
      File "/path/to/conda/environment/lib/python3.10/contextlib.py", line 142, in __exit__
        next(self.gen)
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 826, in incomplete_dir
        shutil.rmtree(dirname)
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 730, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 728, in rmtree
        os.rmdir(path)
    OSError: [Errno 39] Directory not empty: '/path/to/cache/toy-..e05e/1.0.0/...5b4c'
    ```


### Expected behavior

Regenerate the dataset from scratch and reload it.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.2
