For data-only datasets, streaming and non-streaming don't behave the same
COLLABORATOR
Note that we might change the heuristic and create a different config per file, at least in that case.
2022-02-16T15:22:47Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1041632017
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-FgsR
https://api.github.com/repos/huggingface/datasets/issues/comments/1041632017/reactions
2022-02-16T15:22:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1041632017
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
Hi @severo, thanks for reporting.

Yes, this happens because when non-streaming, a cast of all data is done in order to "concatenate" it all into a single dataset (thus the error), while this casting is not done while yielding item by item in streaming mode.

Maybe in streaming mode we should keep the schema (inferred from the first item) and throw an exception if a subsequent item does not conform to the inferred schema?
2022-02-17T06:44:32Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1042625028
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-JTIE
https://api.github.com/repos/huggingface/datasets/issues/comments/1042625028/reactions
2022-02-17T06:44:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1042625028
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
Why do we want to concatenate the files? Is it the expected behavior for most datasets that lack a script and dataset info?
2022-02-17T10:13:09Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1042784344
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-J6BY
https://api.github.com/repos/huggingface/datasets/issues/comments/1042784344/reactions
2022-02-17T10:13:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1042784344
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
These files are two different dataset configurations since they don't share the same schema.

IMO the streaming mode should fail in this case, as @albertvillanova said.

There is one challenge though: inferring the schema from the first example is not robust enough in the general case - especially if some fields are nullable. I guess we can at least make sure that no new columns are added
2022-02-17T15:00:24Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1043041621
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-K41V
https://api.github.com/repos/huggingface/datasets/issues/comments/1043041621/reactions
2022-02-17T15:00:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1043041621
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
OK. So, if we make the streaming also fail, the dataset https://huggingface.co/datasets/huggingface/transformers-metadata will never be [viewable](https://github.com/huggingface/datasets-preview-backend/issues/144) (be it using streaming or fallback to downloading the files), right?

2022-02-17T15:05:08Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1043046541
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-K6CN
https://api.github.com/repos/huggingface/datasets/issues/comments/1043046541/reactions
2022-02-17T15:05:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1043046541
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
Yes, until we have a way for the user to specify explicitly that those two files are different configurations.

We can maybe have some rule to detect this automatically, maybe checking the first line of each file ? That would mean that for dataset of 10,000+ files we would have to verify every single one of them just to know if there is one ore more configurations, so I'm not sure if this is a good idea
2022-02-17T17:50:54Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1043239813
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-LpOF
https://api.github.com/repos/huggingface/datasets/issues/comments/1043239813/reactions
2022-02-17T17:50:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1043239813
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
i think requiring the user to specify that those two files are different configurations is in that case perfectly reasonable.

(Maybe at some point we could however detect this type of case and prompt them to define a config mapping etc)
2022-02-18T22:49:42Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1045301248
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-TggA
https://api.github.com/repos/huggingface/datasets/issues/comments/1045301248/reactions
2022-02-18T22:49:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1045301248
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
COLLABORATOR
OK, so, before closing the issue, what do you think should be done?

> Maybe in streaming mode we should keep the schema (inferred from the first item) and throw an exception if a subsequent item does not conform to the inferred schema?

or nothing?
2022-02-21T10:31:13Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1046724365
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-Y78N
https://api.github.com/repos/huggingface/datasets/issues/comments/1046724365/reactions
2022-02-21T10:31:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1046724365
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
We should at least raise an error if a new sample has column names that are missing, or if it has extra columns. No need to check for the type for now.

I'm in favor of having an error especially because we want to avoid silent issues as much as possible - i.e. when something goes wrong (when schemas don't match or some data are missing) and no errors/warnings are raised.

Consistency between streaming and non-streaming is also important.
2022-02-21T14:24:55Z
https://github.com/huggingface/datasets/issues/3738#issuecomment-1046934692
https://api.github.com/repos/huggingface/datasets/issues/3738
IC_kwDODunzps4-ZvSk
https://api.github.com/repos/huggingface/datasets/issues/comments/1046934692/reactions
2022-02-21T14:24:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1046934692
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
See https://huggingface.co/datasets/huggingface/transformers-metadata: it only contains two JSON files.

In streaming mode, the files are concatenated, and thus the rows might be dictionaries with different keys:

```python
import datasets as ds
iterable_dataset = ds.load_dataset("huggingface/transformers-metadata", split="train", streaming=True);
rows = list(iterable_dataset.take(100))
rows[0]
# {'model_type': 'albert', 'pytorch': True, 'tensorflow': True, 'flax': True, 'processor': 'AutoTokenizer'}
rows[99]
# {'model_class': 'BartModel', 'pipeline_tag': 'feature-extraction', 'auto_class': 'AutoModel'}
```

In normal mode, an exception is thrown:

```python
import datasets as ds
dataset = ds.load_dataset("huggingface/transformers-metadata", split="train");
```

```
ValueError: Couldn't cast
model_class: string
pipeline_tag: string
auto_class: string
to
{'model_type': Value(dtype='string', id=None), 'pytorch': Value(dtype='bool', id=None), 'tensorflow': Value(dtype='bool', id=None), 'flax': Value(dtype='bool', id=None), 'processor': Value(dtype='string', id=None)}
because column names don't match
```