fix: files counted twice in inferred structure
CONTRIBUTOR
I see it creates some errors in the tests.

Another solution if needed is to add something like `data_files = list(set(data_files))` after [this line](https://github.com/huggingface/datasets/blob/8555197a3fe826e98bd0206c2d031c4488c53c5c/src/datasets/data_files.py#L511)
2021-11-21T22:43:08Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-974911077
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46G_Zl
https://api.github.com/repos/huggingface/datasets/issues/comments/974911077/reactions
2021-11-21T22:43:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/974911077
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
MEMBER
Hi ! Thanks for the correction :)

Your change seems right, let me look at the errors and try to fix this
2021-11-22T09:57:34Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-975348577
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46IqNh
https://api.github.com/repos/huggingface/datasets/issues/comments/975348577/reactions
2021-11-22T09:57:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/975348577
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Not sure if it's due to this change but I tested `load_dataset('dalle-mini/encoded-vqgan_imagenet_f16_16384', streaming=True)` and the `validation` set is empty.
2021-11-22T22:47:32Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-975983871
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46LFT_
https://api.github.com/repos/huggingface/datasets/issues/comments/975983871/reactions
2021-11-22T22:47:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/975983871
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
MEMBER
So indeed there was an issue with the patterns `*` and `**/*` that would return some files twice. This issue came from the fact that we were not using the right `glob`.

Indeed we were using `Path.rglob` for local files and `Path.match` for remote files. Since these two methods don't have the same behavior for such patterns, I decided to change that.

In particular, we now use `glob.glob` (same as `fsspec` glob) as a reference for data files resolution from patterns. This is the same as dask for example.

/!\ Here are some behaviors specific to `glob.glob` that are different from Path.glob, Path.match or fnmatch:
- '*' matches only first level files
- '**/*' matches only at least second level files

This way we have a consistent behavior with respect to other python data libraries and there's no overlap anymore between the two patterns.

Some implementations details:

To ensure that we have the same behavior for local files and for files in a remote dataset repository, I decided to use `fsspec` glob for both. This was made possible by implementing the `HfFileSystem` class as a `fsspec` filesystem.

I pushed those changes directly to your PR - I hope you don't mind. I'm still fixing the remaining tests.
Please let me know if that solves your problem, and then we can merge !
2021-11-23T10:27:16Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-976377725
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46Mld9
https://api.github.com/repos/huggingface/datasets/issues/comments/976377725/reactions
2021-11-23T10:38:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976377725
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
There's still an issue with fsspec's glob - I'll take a look this afternoon
2021-11-23T10:56:34Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-976399798
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46Mq22
https://api.github.com/repos/huggingface/datasets/issues/comments/976399798/reactions
2021-11-23T10:56:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976399798
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just found out that actually glob.glob and fsspec glob are different haha
glob.glob needs `**/*` and recursive=True to look into deep subdirectories, while fsspec only requires `**`

I think we can go with fsspec glob for consistency with dask and since it's our main tool for filesystems management
2021-11-23T14:30:43Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-976622234
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46NhKa
https://api.github.com/repos/huggingface/datasets/issues/comments/976622234/reactions
2021-11-23T14:32:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976622234
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
To recap:
```
We use fsspec glob as a reference for data files resolution from patterns.
This is the same as dask for example.

/!\ Here are some behaviors specific to fsspec glob that are different from glob.glob, Path.glob, Path.match or fnmatch:
- '*' matches only first level items
- '**' matches all items
- '**/*' matches all at least second level items

More generally:
- `*`` matches any character except a forward-slash (to match just the file or directory name)
- `**`` matches any character including a forward-slash /
```
2021-11-23T14:43:59Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-976650514
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46NoES
https://api.github.com/repos/huggingface/datasets/issues/comments/976650514/reactions
2021-11-23T14:43:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976650514
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
lol Windows… Maybe `Pathlib` for the tests?

I tested streaming a repo and it worked perfectly now!
2021-11-23T15:15:58Z
https://github.com/huggingface/datasets/pull/3309#issuecomment-976696206
https://api.github.com/repos/huggingface/datasets/issues/3309
IC_kwDODunzps46NzOO
https://api.github.com/repos/huggingface/datasets/issues/comments/976696206/reactions
2021-11-23T15:15:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976696206
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
Files were counted twice in a structure like:
```
my_dataset_local_path/
├── README.md
└── data/
    ├── train/
    │   ├── shard_0.csv
    │   ├── shard_1.csv
    │   ├── shard_2.csv
    │   └── shard_3.csv
    └── valid/
        ├── shard_0.csv
        └── shard_1.csv
```

The reason is that they were matching both `*train*/*` and `*train*/**/*`.

This PR fixes it. @lhoestq 