Added method to remove excess nesting in a DatasetDict
MEMBER
Hi ! I think the issue you linked is closed and suggests to use `remove_columns`.

Moreover if you end up with a dataset with an unnecessarily nested data, please modify your processing functions to not output nested data, or use `map(..., batched=True)` if you function take batches as input
2022-07-18T09:05:09Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1186948980
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5Gv2d0
https://api.github.com/repos/huggingface/datasets/issues/comments/1186948980/reactions
2022-07-18T09:05:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1186948980
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq , you are right about the issues this pull has steered beyond that issue. I created this [colab notebook](https://colab.research.google.com/drive/16aLu6QrDSV_aUYRdpufl5E4iS08qkUGj?usp=sharing) to present the error. I tried using batch and that won't resolve it either. I'm looking into that error right now.
2022-07-18T17:15:45Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1187810986
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5GzI6q
https://api.github.com/repos/huggingface/datasets/issues/comments/1187810986/reactions
2022-07-18T17:15:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187810986
https://avatars.githubusercontent.com/u/37946988?v=4
https://api.github.com/users/CakeCrusher/events{/privacy}
https://api.github.com/users/CakeCrusher/followers
https://api.github.com/users/CakeCrusher/following{/other_user}
https://api.github.com/users/CakeCrusher/gists{/gist_id}
https://github.com/CakeCrusher
CakeCrusher
MDQ6VXNlcjM3OTQ2OTg4
https://api.github.com/users/CakeCrusher/orgs
https://api.github.com/users/CakeCrusher/received_events
https://api.github.com/users/CakeCrusher/repos
https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}
https://api.github.com/users/CakeCrusher/subscriptions
User
https://api.github.com/users/CakeCrusher
public
MEMBER
I think you just need to pass one example at a time to your tokenizer, this way you don't end up with nested data:
```python

def preprocessFunction(row):
  collatedContext = tokenizer.eos_token.join([row["context"+str(i+1)] for i in range(int(AMT_OF_CONTEXT))])
  response = row["response"]
  tokenizedContext = tokenizer(
    collatedContext, max_length=max_context_length, truncation=True  # don't pass as a list here
  )
  with tokenizer.as_target_tokenizer():
    tokenized_response = tokenizer(
      response, max_length=max_response_length, truncation=True  # don't pass a a list here
    )
  tokenizedContext["labels"] = tokenized_response["input_ids"]
  return tokenizedContext
```
2022-07-18T17:23:44Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1187843577
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5GzQ35
https://api.github.com/repos/huggingface/datasets/issues/comments/1187843577/reactions
2022-07-18T17:23:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187843577
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes that is correct, the purpose of this pull is to advise of a more general solution like with `def remove_excess_nesting(self)` or maybe automate the solution (stas00 advised not to automate it as it could "not be backwards compatible").
2022-07-18T17:38:01Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1187898101
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5GzeL1
https://api.github.com/repos/huggingface/datasets/issues/comments/1187898101/reactions
2022-07-18T17:38:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187898101
https://avatars.githubusercontent.com/u/37946988?v=4
https://api.github.com/users/CakeCrusher/events{/privacy}
https://api.github.com/users/CakeCrusher/followers
https://api.github.com/users/CakeCrusher/following{/other_user}
https://api.github.com/users/CakeCrusher/gists{/gist_id}
https://github.com/CakeCrusher
CakeCrusher
MDQ6VXNlcjM3OTQ2OTg4
https://api.github.com/users/CakeCrusher/orgs
https://api.github.com/users/CakeCrusher/received_events
https://api.github.com/users/CakeCrusher/repos
https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}
https://api.github.com/users/CakeCrusher/subscriptions
User
https://api.github.com/users/CakeCrusher
public
MEMBER
I'm not sure I understand how having `remove_excess_nesting` would make more sense than just fixing the preprocessFunction to simply not return nested samples, can you elaborate ?
2022-07-18T17:44:14Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1187915450
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5Gzia6
https://api.github.com/repos/huggingface/datasets/issues/comments/1187915450/reactions
2022-07-18T17:44:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187915450
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Figuring out the issue can be a bit difficult to figure out. Only until I added batch does it make a little more sense with the error

> sequence item 0: expected str instance, list found

but batch was never intended.

When you run the colab you will notice that only until collating do you learn there is this error. So i figured it would be better to address it during at the `DatasetDict` level.
I think it would be ideal if the user could be notified at the preprocess function.
2022-07-18T17:56:37Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1187955168
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5GzsHg
https://api.github.com/repos/huggingface/datasets/issues/comments/1187955168/reactions
2022-07-18T17:56:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187955168
https://avatars.githubusercontent.com/u/37946988?v=4
https://api.github.com/users/CakeCrusher/events{/privacy}
https://api.github.com/users/CakeCrusher/followers
https://api.github.com/users/CakeCrusher/following{/other_user}
https://api.github.com/users/CakeCrusher/gists{/gist_id}
https://github.com/CakeCrusher
CakeCrusher
MDQ6VXNlcjM3OTQ2OTg4
https://api.github.com/users/CakeCrusher/orgs
https://api.github.com/users/CakeCrusher/received_events
https://api.github.com/users/CakeCrusher/repos
https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}
https://api.github.com/users/CakeCrusher/subscriptions
User
https://api.github.com/users/CakeCrusher
public
NONE
I'm not arguing that `remove_excess_nesting` is the right solution but what I aim to address is dealing with unnecessary nesting as early as possible.
2022-07-18T17:59:15Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1187965781
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5GzutV
https://api.github.com/repos/huggingface/datasets/issues/comments/1187965781/reactions
2022-07-18T17:59:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1187965781
https://avatars.githubusercontent.com/u/37946988?v=4
https://api.github.com/users/CakeCrusher/events{/privacy}
https://api.github.com/users/CakeCrusher/followers
https://api.github.com/users/CakeCrusher/following{/other_user}
https://api.github.com/users/CakeCrusher/gists{/gist_id}
https://github.com/CakeCrusher
CakeCrusher
MDQ6VXNlcjM3OTQ2OTg4
https://api.github.com/users/CakeCrusher/orgs
https://api.github.com/users/CakeCrusher/received_events
https://api.github.com/users/CakeCrusher/repos
https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}
https://api.github.com/users/CakeCrusher/subscriptions
User
https://api.github.com/users/CakeCrusher
public
MEMBER
> When you run the colab you will notice that only until collating do you learn there is this error.

I think users can just check the `dataset.features` and they would notice that the data are nested
```python
{
   'input_ids': Sequence(Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), length=-1, id=None)
   ...
}
```

Sometime nested data are intentional, so you can't know in advance if it's a user's mistake or something planned.
2022-07-19T13:21:17Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1189046663
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5G32mH
https://api.github.com/repos/huggingface/datasets/issues/comments/1189046663/reactions
2022-07-19T13:21:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1189046663
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes, I understand, it could be intentional and only the collator has problems with it. So, it is not worth handling it any differently in any other non-erroneous data. 

That being said do you think there is any use for the `remove_excess_nesting` method? Or maybe it should be applied in a different way? If not feel free to close this PR. 
2022-07-20T19:25:43Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1190667318
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5G-CQ2
https://api.github.com/repos/huggingface/datasets/issues/comments/1190667318/reactions
2022-07-20T19:25:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1190667318
https://avatars.githubusercontent.com/u/37946988?v=4
https://api.github.com/users/CakeCrusher/events{/privacy}
https://api.github.com/users/CakeCrusher/followers
https://api.github.com/users/CakeCrusher/following{/other_user}
https://api.github.com/users/CakeCrusher/gists{/gist_id}
https://github.com/CakeCrusher
CakeCrusher
MDQ6VXNlcjM3OTQ2OTg4
https://api.github.com/users/CakeCrusher/orgs
https://api.github.com/users/CakeCrusher/received_events
https://api.github.com/users/CakeCrusher/repos
https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}
https://api.github.com/users/CakeCrusher/subscriptions
User
https://api.github.com/users/CakeCrusher
public
MEMBER
I think users can write it and use `map` themselves if needed, it is pretty straightforward to implement.

I'm closing this PR if you don't mind, and thank you for the discussion :)
2022-07-21T10:55:02Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1191341141
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5HAmxV
https://api.github.com/repos/huggingface/datasets/issues/comments/1191341141/reactions
2022-07-21T10:55:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191341141
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
No problem @lhoestq , thanks for walking me through it.
2022-07-21T15:55:25Z
https://github.com/huggingface/datasets/pull/4679#issuecomment-1191660828
https://api.github.com/repos/huggingface/datasets/issues/4679
IC_kwDODunzps5HB00c
https://api.github.com/repos/huggingface/datasets/issues/comments/1191660828/reactions
2022-07-21T15:55:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191660828
https://avatars.githubusercontent.com/u/37946988?v=4
https://api.github.com/users/CakeCrusher/events{/privacy}
https://api.github.com/users/CakeCrusher/followers
https://api.github.com/users/CakeCrusher/following{/other_user}
https://api.github.com/users/CakeCrusher/gists{/gist_id}
https://github.com/CakeCrusher
CakeCrusher
MDQ6VXNlcjM3OTQ2OTg4
https://api.github.com/users/CakeCrusher/orgs
https://api.github.com/users/CakeCrusher/received_events
https://api.github.com/users/CakeCrusher/repos
https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}
https://api.github.com/users/CakeCrusher/subscriptions
User
https://api.github.com/users/CakeCrusher
public
Added the ability for a DatasetDict to remove additional nested layers within its features to avoid conflicts when collating. It is meant to accompany [this PR](https://github.com/huggingface/transformers/pull/18119) to resolve the same issue [#15505](https://github.com/huggingface/transformers/issues/15505).

@stas00 @lhoestq 