WebDataset with different prefixes are unsupported
MEMBER
Since `datasets` uses is built on Arrow to store the data, it requires each sample to have the same columns.

This can be fixed by specifyign in advance the name of all the possible columns in the `dataset_info` in YAML, and missing values will be `None`
2024-07-22T13:50:08Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2243014311
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6Fsa6n
https://api.github.com/repos/huggingface/datasets/issues/comments/2243014311/reactions
2024-07-22T13:50:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2243014311
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks. This currently doesn't work for WebDataset because there's no `BuilderConfig` with `features` and in turn `_info` is missing `features=self.config.features`. I'll prepare a PR to fix this.

Note it may be useful to add the [expected format of `features`](https://github.com/huggingface/datasets/blob/16fa4421f44b22bbbc607f379a93f45af468d1fc/src/datasets/features/features.py#L1757) to the documentation for [`Builder Parameters`](https://huggingface.co/docs/datasets/repository_structure#builder-parameters).

2024-07-22T15:31:54Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2243247211
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6FtTxr
https://api.github.com/repos/huggingface/datasets/issues/comments/2243247211/reactions
2024-07-22T15:31:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2243247211
https://avatars.githubusercontent.com/u/106811348?v=4
https://api.github.com/users/hlky/events{/privacy}
https://api.github.com/users/hlky/followers
https://api.github.com/users/hlky/following{/other_user}
https://api.github.com/users/hlky/gists{/gist_id}
https://github.com/hlky
hlky
U_kgDOBl3P1A
https://api.github.com/users/hlky/orgs
https://api.github.com/users/hlky/received_events
https://api.github.com/users/hlky/repos
https://api.github.com/users/hlky/starred{/owner}{/repo}
https://api.github.com/users/hlky/subscriptions
User
https://api.github.com/users/hlky
public
MEMBER
Oh good catch ! thanks

> Note it may be useful to add the [expected format of features](https://github.com/huggingface/datasets/blob/16fa4421f44b22bbbc607f379a93f45af468d1fc/src/datasets/features/features.py#L1757) to the documentation for [Buil](https://huggingface.co/docs/datasets/repository_structure#builder-parameters)

Good idea, let me open a PR
2024-07-22T15:35:28Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2243254910
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6FtVp-
https://api.github.com/repos/huggingface/datasets/issues/comments/2243254910/reactions
2024-07-22T15:35:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2243254910
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
#7060 
2024-07-22T15:42:21Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2243269238
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6FtZJ2
https://api.github.com/repos/huggingface/datasets/issues/comments/2243269238/reactions
2024-07-22T15:42:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2243269238
https://avatars.githubusercontent.com/u/106811348?v=4
https://api.github.com/users/hlky/events{/privacy}
https://api.github.com/users/hlky/followers
https://api.github.com/users/hlky/following{/other_user}
https://api.github.com/users/hlky/gists{/gist_id}
https://github.com/hlky
hlky
U_kgDOBl3P1A
https://api.github.com/users/hlky/orgs
https://api.github.com/users/hlky/received_events
https://api.github.com/users/hlky/repos
https://api.github.com/users/hlky/starred{/owner}{/repo}
https://api.github.com/users/hlky/subscriptions
User
https://api.github.com/users/hlky
public
MEMBER
Actually I just tried with `datasets` on the `main` branch and having `features` defined in `dataset_info` worked for me

```python
>>> list(load_dataset("/Users/quentinlhoest/tmp", streaming=True, split="train"))
[{'txt': 'hello there\n', 'other': None}]
```
where `tmp` contains data.tar with "hello there\n" in a text file and the README.md:
```
---
dataset_info:
  features:
  - name: txt
    dtype: string
  - name: other
    dtype: string
---

This is a dataset card
```

What error did you get when you tried to specify the columns in `dataset_info` ?
2024-07-23T12:59:17Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2245182973
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6F0sX9
https://api.github.com/repos/huggingface/datasets/issues/comments/2245182973/reactions
2024-07-23T12:59:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2245182973
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
If you review the changes in #7060 you'll note that `features` are not passed to `DatasetInfo`.

In your case the features are being extracted by [this code](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/webdataset/webdataset.py#L72-L98).

Try with the `Steps to reproduce the bug`. It's the same error mentioned in `Describe the bug` because `features` are not passed to `DatasetInfo`.

`features` are [not used](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/builder.py#L365-L366) when the `BuilderConfig` has no `features` attribute. `WebDataset` uses the default [`BuilderConfig`](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/builder.py#L101-L124).

There is a [warning](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/load.py#L640-L648) that `features` are ignored.

Note that as mentioned in `Describe the bug` this could also be resolved by removing the check [here](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/webdataset/webdataset.py#L76-L80) because Arrow actually handles this itself, Arrow sets any missing fields to `None`, at least in my case.
2024-07-23T13:08:46Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2245207027
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6F0yPz
https://api.github.com/repos/huggingface/datasets/issues/comments/2245207027/reactions
2024-07-23T13:19:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2245207027
https://avatars.githubusercontent.com/u/106811348?v=4
https://api.github.com/users/hlky/events{/privacy}
https://api.github.com/users/hlky/followers
https://api.github.com/users/hlky/following{/other_user}
https://api.github.com/users/hlky/gists{/gist_id}
https://github.com/hlky
hlky
U_kgDOBl3P1A
https://api.github.com/users/hlky/orgs
https://api.github.com/users/hlky/received_events
https://api.github.com/users/hlky/repos
https://api.github.com/users/hlky/starred{/owner}{/repo}
https://api.github.com/users/hlky/subscriptions
User
https://api.github.com/users/hlky
public
NONE
Note for anyone else who encounters this issue, every dataset type except folder-based types supported features in the [documented](https://huggingface.co/docs/datasets/repository_structure#builder-parameters) manner; [Arrow](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/arrow/arrow.py#L15-L21), [csv](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/csv/csv.py#L25-L68), [generator](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/generator/generator.py#L8-L19), [json](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/json/json.py#L42-L52), [pandas](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/pandas/pandas.py#L14-L20), [parquet](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/parquet/parquet.py#L16-L24), [spark](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/spark/spark.py#L31-L37), [sql](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/sql/sql.py#L24-L35) and [text](https://github.com/huggingface/datasets/blob/e83d6fa574710fcb44e341087239d2687183f62b/src/datasets/packaged_modules/text/text.py#L18-L27). `WebDataset` is different and requires [`dataset_info` which is vaguely documented](https://huggingface.co/docs/datasets/dataset_script#optional-generate-dataset-metadata) under dataset loading scripts.
2024-07-23T13:58:00Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2245334124
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6F1RRs
https://api.github.com/repos/huggingface/datasets/issues/comments/2245334124/reactions
2024-07-23T13:58:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2245334124
https://avatars.githubusercontent.com/u/106811348?v=4
https://api.github.com/users/hlky/events{/privacy}
https://api.github.com/users/hlky/followers
https://api.github.com/users/hlky/following{/other_user}
https://api.github.com/users/hlky/gists{/gist_id}
https://github.com/hlky
hlky
U_kgDOBl3P1A
https://api.github.com/users/hlky/orgs
https://api.github.com/users/hlky/received_events
https://api.github.com/users/hlky/repos
https://api.github.com/users/hlky/starred{/owner}{/repo}
https://api.github.com/users/hlky/subscriptions
User
https://api.github.com/users/hlky
public
MEMBER
Thanks for explaining. I see the Dataset Viewer is still failing - I'll update `datasets` in the Viewer to fix this
2024-07-24T13:26:29Z
https://github.com/huggingface/datasets/issues/7055#issuecomment-2247933680
https://api.github.com/repos/huggingface/datasets/issues/7055
IC_kwDODunzps6F_L7w
https://api.github.com/repos/huggingface/datasets/issues/comments/2247933680/reactions
2024-07-24T13:26:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2247933680
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

Consider a WebDataset with multiple images for each item where the number of images may vary: [example](https://huggingface.co/datasets/bigdata-pw/fashion-150k)

Due to this [code](https://github.com/huggingface/datasets/blob/87f4c2088854ff33e817e724e75179e9975c1b02/src/datasets/packaged_modules/webdataset/webdataset.py#L76-L80) an error is given.
```
The TAR archives of the dataset should be in WebDataset format, but the files in the archive don't share the same prefix or the same types.
```

The purpose of this check is unclear because PyArrow supports different keys.

Removing the check allows the dataset to be loaded and there's no issue when iterating through the dataset.

```
>>> from datasets import load_dataset
>>> path = "shards/*.tar"
>>> dataset = load_dataset("webdataset", data_files={"train": path}, split="train", streaming=True)
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 56458.93it/s]
>>> dataset
IterableDataset({
    features: ['__key__', '__url__', '1.jpg', '2.jpg', '3.jpg', '4.jpg', 'json'],
    n_shards: 152
})
```

### Steps to reproduce the bug

```python
from datasets import load_dataset
load_dataset("bigdata-pw/fashion-150k")
```

### Expected behavior

Dataset loads without error

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.14.0-467.el9.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.19
- `huggingface_hub` version: 0.23.4
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0