Make packaged builders skip non-supported file formats
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5850). All of your documentation changes will be reflected on that endpoint.
2023-05-12T13:56:51Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1545790902
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cIuW2
https://api.github.com/repos/huggingface/datasets/issues/comments/1545790902/reactions
2023-05-12T13:56:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1545790902
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Good idea. @mariosasko!!!

Please note that before this PR, the files are not evenly distributed for archives: `_generate_examples` gets a list of iterators, one for each archive (uncompressed to a directory).
2023-05-12T17:14:38Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1546048916
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cJtWU
https://api.github.com/repos/huggingface/datasets/issues/comments/1546048916/reactions
2023-05-12T17:20:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1546048916
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
This change could create silent problems when loading files with extensions that are not listed here. For example

```python
load_dataset("text", data_files=["20230515.log"])
```

wouldn't even log anything to say that the file was ignored.

Maybe it's possible to do this at data files patterns resolution ?

e.g. in get_data_patterns_in_dataset_repository / get_data_patterns_locally we could return patterns that include the most common extension
2023-05-15T10:12:15Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1547577229
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cPieN
https://api.github.com/repos/huggingface/datasets/issues/comments/1547577229/reactions
2023-05-15T10:12:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1547577229
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq the issue you evoke (.log files skipped by text builder if .log is not added to .txt as supported extension) persists whether you perform the skip at the pattern resolution or in the builder itself.

The solution is to add the .log extension (besides the .txt) as supported by text, independently of where we perform the skip (at pattern resolution or in the builder itself).

Additionally, at the time we call for pattern resolution, we do not know the builder class yet, so that we cannot pass specific file extensions. First we call data files pattern resolution, and afterwards we call `infer_module_for_data_files` and then know the builder class.
2023-05-16T09:17:24Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1549300668
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cWHO8
https://api.github.com/repos/huggingface/datasets/issues/comments/1549300668/reactions
2023-05-16T09:18:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549300668
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
> @lhoestq the issue you evoke (.log files skipped by text builder if .log is not added to .txt as supported extension) persists whether you perform the skip at the pattern resolution or in the builder itself.

No I simply think it's a bad breaking change to not support

```python
load_dataset("<builder_name>", data_files=["path/to/file_with_unknown_or_no_extension"])
# or
load_dataset("<builder_name>", data_files=["https://url.to/file_with_unknown_or_no_extension"])
```

Idk if it's the easiest solution, but maybe it's possible to do the change only when inferring the patterns of dataset repositories. This should avoid this breaking change.

For example it could do something like that in `get_data_patterns_locally`

```python
    Input:

        my_dataset_repository/
        ├── README.md
        ├── banner.png
        ├── data0.csv
        ├── data1.csv
        └── data2.csv

    Output:

        {"train": ["**.csv"]}
```

instead of 

```python
    Output:

        {"train": ["**"]}
```
2023-05-16T13:02:17Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1549622184
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cXVuo
https://api.github.com/repos/huggingface/datasets/issues/comments/1549622184/reactions
2023-05-16T13:02:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549622184
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
I agree with @lhoestq - it should still be possible to request parsing a file with a specific builder even if the file's extension is "invalid" for the builder, and only ignore non-supported file formats when inferring the patterns.
2023-05-16T14:51:43Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1549830365
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cYIjd
https://api.github.com/repos/huggingface/datasets/issues/comments/1549830365/reactions
2023-05-16T14:51:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549830365
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Therefore, if I understand correctly, what you suggest is:
- if the user passes a packaged builder to `load_dataset` (e.g. `load_dataset("csv",...`), then the *passed* `data_files` should not be filtered to remove unsupported extensions. No breaking change in this case
- if the user passes a no-script repo/folder to `load_dataset` (e.g. `load_dataset("my_dataset_repository",...`), then the *inferred* data files should be filtered to remove the extensions that are not supported by the inferred module name builder
  - if the user passes `data_files` as well, then I guess these should not be filtered, to avoid any breaking change as in the first case above
2023-05-17T05:05:49Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1550740126
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cbmqe
https://api.github.com/repos/huggingface/datasets/issues/comments/1550740126/reactions
2023-05-17T05:14:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1550740126
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yes that would be ideal imo !
2023-05-17T12:08:25Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1551276380
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5cdplc
https://api.github.com/repos/huggingface/datasets/issues/comments/1551276380/reactions
2023-05-17T12:08:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1551276380
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I think this now fulfills all the requirements.
2023-06-05T11:15:35Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1576599610
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5d-QA6
https://api.github.com/repos/huggingface/datasets/issues/comments/1576599610/reactions
2023-06-05T11:15:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1576599610
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I find it a bit confusing to still be able to pass data_files that are going to be silently ignored based on the value of `only_supported_extensions`. My suggestion was to have the right data files pattern, not to filter a posteriori (sorry if my last message was confusing).

Having the right data files pattern would also allow users to inspect what's actually being loaded with
```
load_dataset_builder(...).config.data_files
```
and it would list exactly what data files are used.
2023-06-07T09:19:47Z
https://github.com/huggingface/datasets/pull/5850#issuecomment-1580280565
https://api.github.com/repos/huggingface/datasets/issues/5850
IC_kwDODunzps5eMSr1
https://api.github.com/repos/huggingface/datasets/issues/comments/1580280565/reactions
2023-06-07T09:19:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1580280565
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This PR makes packaged builders skip non-supported file formats:
- Csv builder skips non-CSV files
- Analogously for the other builders

Fix #5849.