Identify which shard data belongs to
CONTRIBUTOR
Independently of this I think it raises the need to allow multiprocessing during streaming so that we get samples from multiple shards in one batch.
2021-10-09T20:24:17Z
https://github.com/huggingface/datasets/issues/3048#issuecomment-939356949
https://api.github.com/repos/huggingface/datasets/issues/3048
IC_kwDODunzps43_XMV
https://api.github.com/repos/huggingface/datasets/issues/comments/939356949/reactions
2021-10-09T20:24:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/939356949
https://avatars.githubusercontent.com/u/715491?v=4
https://api.github.com/users/borisdayma/events{/privacy}
https://api.github.com/users/borisdayma/followers
https://api.github.com/users/borisdayma/following{/other_user}
https://api.github.com/users/borisdayma/gists{/gist_id}
https://github.com/borisdayma
borisdayma
MDQ6VXNlcjcxNTQ5MQ==
https://api.github.com/users/borisdayma/orgs
https://api.github.com/users/borisdayma/received_events
https://api.github.com/users/borisdayma/repos
https://api.github.com/users/borisdayma/starred{/owner}{/repo}
https://api.github.com/users/borisdayma/subscriptions
User
https://api.github.com/users/borisdayma
public
**Is your feature request related to a problem? Please describe.**

I'm training on a large dataset made of multiple sub-datasets.
During training I can observe some jumps in loss which may correspond to different shards.

![image](https://user-images.githubusercontent.com/715491/136668758-521263aa-a9b2-4ad2-8d22-060b6bf86a1c.png)

My suspicion is that either:
* some of the sub-datasets are harder for the model than others
* some of the sub-datasets are not formatted properly

I'd like to identify which shards correspond to those jumps.

**Describe the solution you'd like**

It would be nice to have a key associated to each data sample or data batch containing details on where the data comes from (shard idx + item idx within the shard).
This should be supported both in local and streaming mode.

**Describe alternatives you've considered**
A fix would be for me to add myself details (shard id, sample id) as part of each data sample.
The inconvenient is that it requires users to process/reupload every dataset when they need this feature.