data configuration hash suffix depends on uncanonicalized data_dir
CONTRIBUTOR
It could even use `os.path.realpath` to resolve symlinks.
2023-05-16T19:07:58Z
https://github.com/huggingface/datasets/issues/5871#issuecomment-1550211454
https://api.github.com/repos/huggingface/datasets/issues/5871
IC_kwDODunzps5cZll-
https://api.github.com/repos/huggingface/datasets/issues/comments/1550211454/reactions
2023-05-16T19:07:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1550211454
https://avatars.githubusercontent.com/u/5044802?v=4
https://api.github.com/users/kylrth/events{/privacy}
https://api.github.com/users/kylrth/followers
https://api.github.com/users/kylrth/following{/other_user}
https://api.github.com/users/kylrth/gists{/gist_id}
https://github.com/kylrth
kylrth
MDQ6VXNlcjUwNDQ4MDI=
https://api.github.com/users/kylrth/orgs
https://api.github.com/users/kylrth/received_events
https://api.github.com/users/kylrth/repos
https://api.github.com/users/kylrth/starred{/owner}{/repo}
https://api.github.com/users/kylrth/subscriptions
User
https://api.github.com/users/kylrth
public
COLLABORATOR
Indeed, it makes sense to normalize `data_dir`. Feel free to submit a PR (this can be "fixed" [here](https://github.com/huggingface/datasets/blob/89f775226321ba94e5bf4670a323c0fb44f5f65c/src/datasets/builder.py#L173))
2023-05-25T15:24:38Z
https://github.com/huggingface/datasets/issues/5871#issuecomment-1563099878
https://api.github.com/repos/huggingface/datasets/issues/5871
IC_kwDODunzps5dKwLm
https://api.github.com/repos/huggingface/datasets/issues/comments/1563099878/reactions
2023-05-25T15:24:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1563099878
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
#self-assign
2023-05-25T17:42:29Z
https://github.com/huggingface/datasets/issues/5871#issuecomment-1563278516
https://api.github.com/repos/huggingface/datasets/issues/5871
IC_kwDODunzps5dLby0
https://api.github.com/repos/huggingface/datasets/issues/comments/1563278516/reactions
2023-05-25T17:42:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1563278516
https://avatars.githubusercontent.com/u/5044802?v=4
https://api.github.com/users/kylrth/events{/privacy}
https://api.github.com/users/kylrth/followers
https://api.github.com/users/kylrth/following{/other_user}
https://api.github.com/users/kylrth/gists{/gist_id}
https://github.com/kylrth
kylrth
MDQ6VXNlcjUwNDQ4MDI=
https://api.github.com/users/kylrth/orgs
https://api.github.com/users/kylrth/received_events
https://api.github.com/users/kylrth/repos
https://api.github.com/users/kylrth/starred{/owner}{/repo}
https://api.github.com/users/kylrth/subscriptions
User
https://api.github.com/users/kylrth
public
### Describe the bug

I am working with the `recipe_nlg` dataset, which requires manual download. Once it's downloaded, I've noticed that the hash in the custom data configuration is different if I add a trailing `/` to my `data_dir`. It took me a while to notice that the hashes were different, and to understand that that was the cause of my dataset being processed anew instead of the cached version being used.

### Steps to reproduce the bug

1. Follow the steps to manually download the `recipe_nlg` dataset to `/data/recipenlg`.
2. Load it using `load_dataset`, once without a trailing slash and once with one:

    ```python
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg")
    Using custom data configuration default-082278caeea85765
    Downloading and preparing dataset recipe_nlg/default to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Dataset recipe_nlg downloaded and prepared to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74. Subsequent calls will reuse this data.
    100%|███████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.10s/it]
    DatasetDict({
        train: Dataset({
            features: ['id', 'title', 'ingredients', 'directions', 'link', 'source', 'ner'],
            num_rows: 2231142
        })
    })
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg/")
    Using custom data configuration default-83e87680785d0493
    Downloading and preparing dataset recipe_nlg/default to /home/user/.cache/huggingface/datasets/recipe_nlg/default-83e87680785d0493/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Generating train split:   1%|          | 12701/2231142 [00:04<13:15, 2790.25 examples/s
    ^C
    ```

3. Observe that the hash suffix in the custom data configuration changes due to the altered string.

### Expected behavior

I think I would expect the hash to remain constant if it actually points to the same location on disk. I would expect the use of `os.path.normpath` to canonicalize the paths.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-147-generic-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.2