equal operation to perform unbatch for huggingface datasets 
NONE
Hi @lhoestq 
Maybe this is clearer to explain like this, currently map function, map one example to "one" modified one, lets assume we want to map one example to "multiple" examples, in which we do not know in advance how many examples they would be per each entry. I greatly appreciate telling me how I can handle this operation, thanks a lot
2021-08-06T19:57:56Z
https://github.com/huggingface/datasets/issues/2767#issuecomment-894487384
https://api.github.com/repos/huggingface/datasets/issues/2767
IC_kwDODunzps41UMtY
https://api.github.com/repos/huggingface/datasets/issues/comments/894487384/reactions
2021-08-06T19:58:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/894487384
https://avatars.githubusercontent.com/u/79288051?v=4
https://api.github.com/users/dorooddorood606/events{/privacy}
https://api.github.com/users/dorooddorood606/followers
https://api.github.com/users/dorooddorood606/following{/other_user}
https://api.github.com/users/dorooddorood606/gists{/gist_id}
https://github.com/dorooddorood606
dorooddorood606
MDQ6VXNlcjc5Mjg4MDUx
https://api.github.com/users/dorooddorood606/orgs
https://api.github.com/users/dorooddorood606/received_events
https://api.github.com/users/dorooddorood606/repos
https://api.github.com/users/dorooddorood606/starred{/owner}{/repo}
https://api.github.com/users/dorooddorood606/subscriptions
User
https://api.github.com/users/dorooddorood606
public
NONE
Hi,
this is also my question on how to perform similar operation as "unbatch" in tensorflow in great huggingface dataset library. 
thanks.
2021-08-07T09:22:39Z
https://github.com/huggingface/datasets/issues/2767#issuecomment-894629984
https://api.github.com/repos/huggingface/datasets/issues/2767
IC_kwDODunzps41Uvhg
https://api.github.com/repos/huggingface/datasets/issues/comments/894629984/reactions
2021-08-07T09:22:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/894629984
https://avatars.githubusercontent.com/u/59409879?v=4
https://api.github.com/users/jackfeinmann5/events{/privacy}
https://api.github.com/users/jackfeinmann5/followers
https://api.github.com/users/jackfeinmann5/following{/other_user}
https://api.github.com/users/jackfeinmann5/gists{/gist_id}
https://github.com/jackfeinmann5
jackfeinmann5
MDQ6VXNlcjU5NDA5ODc5
https://api.github.com/users/jackfeinmann5/orgs
https://api.github.com/users/jackfeinmann5/received_events
https://api.github.com/users/jackfeinmann5/repos
https://api.github.com/users/jackfeinmann5/starred{/owner}{/repo}
https://api.github.com/users/jackfeinmann5/subscriptions
User
https://api.github.com/users/jackfeinmann5
public
COLLABORATOR
Hi,

`Dataset.map` in the batched mode allows you to map a single row to multiple rows. So to perform "unbatch", you can do the following:
```python
import collections

def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {ex['passage']}"
        new_batch["inputs"].extend([inputs] * len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"])
    return new_batch

dset = dset.map(unbatch, batched=True, remove_columns=dset.column_names)
```
2021-08-07T16:56:00Z
https://github.com/huggingface/datasets/issues/2767#issuecomment-894678553
https://api.github.com/repos/huggingface/datasets/issues/2767
IC_kwDODunzps41U7YZ
https://api.github.com/repos/huggingface/datasets/issues/comments/894678553/reactions
2021-08-07T16:56:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/894678553
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Dear @mariosasko 
First, thank you very much for coming back to me on this, I appreciate it a lot. I tried this solution,  I am getting errors, do you mind
giving me one test example to be able to run your code, to understand better the format of the inputs to your function?
in this function https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L952 they copy each example to the number of "answers", do you mean one should not do the copying part and use directly your function? 


thank you very much for your help and time.
2021-08-07T18:58:11Z
https://github.com/huggingface/datasets/issues/2767#issuecomment-894693651
https://api.github.com/repos/huggingface/datasets/issues/2767
IC_kwDODunzps41U_ET
https://api.github.com/repos/huggingface/datasets/issues/comments/894693651/reactions
2021-08-07T19:08:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/894693651
https://avatars.githubusercontent.com/u/79288051?v=4
https://api.github.com/users/dorooddorood606/events{/privacy}
https://api.github.com/users/dorooddorood606/followers
https://api.github.com/users/dorooddorood606/following{/other_user}
https://api.github.com/users/dorooddorood606/gists{/gist_id}
https://github.com/dorooddorood606
dorooddorood606
MDQ6VXNlcjc5Mjg4MDUx
https://api.github.com/users/dorooddorood606/orgs
https://api.github.com/users/dorooddorood606/received_events
https://api.github.com/users/dorooddorood606/repos
https://api.github.com/users/dorooddorood606/starred{/owner}{/repo}
https://api.github.com/users/dorooddorood606/subscriptions
User
https://api.github.com/users/dorooddorood606
public
NONE
Hi @mariosasko 
I think finally I got this, I think you mean to do things in one step, here is the full example for completeness:

```
def unbatch(batch):
    new_batch = collections.defaultdict(list)
    keys = batch.keys()
    for values in zip(*batch.values()):
        ex = {k: v for k, v in zip(keys, values)}
        # updates the passage.
        passage = ex['passage']
        passage = re.sub(r'(\.|\?|\!|\"|\')\n@highlight\n', r'\1 ', passage)
        passage = re.sub(r'\n@highlight\n', '. ', passage)
        inputs = f"record query: {ex['query']} entities: {', '.join(ex['entities'])} passage: {passage}"
        # duplicates the samples based on  number of answers.
        num_answers = len(ex["answers"])
        num_duplicates = np.maximum(1, num_answers)
        new_batch["inputs"].extend([inputs] * num_duplicates) #len(ex["answers"]))
        new_batch["targets"].extend(ex["answers"] if num_answers > 0 else ["<unk>"])
    return new_batch

data = datasets.load_dataset('super_glue', 'record', split="train", script_version="master")
data = data.map(unbatch, batched=True, remove_columns=data.column_names)
```

Thanks a lot again, this was a super great way to do it.
2021-08-07T19:56:20Z
https://github.com/huggingface/datasets/issues/2767#issuecomment-894700413
https://api.github.com/repos/huggingface/datasets/issues/2767
IC_kwDODunzps41VAt9
https://api.github.com/repos/huggingface/datasets/issues/comments/894700413/reactions
2021-08-07T19:56:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/894700413
https://avatars.githubusercontent.com/u/79288051?v=4
https://api.github.com/users/dorooddorood606/events{/privacy}
https://api.github.com/users/dorooddorood606/followers
https://api.github.com/users/dorooddorood606/following{/other_user}
https://api.github.com/users/dorooddorood606/gists{/gist_id}
https://github.com/dorooddorood606
dorooddorood606
MDQ6VXNlcjc5Mjg4MDUx
https://api.github.com/users/dorooddorood606/orgs
https://api.github.com/users/dorooddorood606/received_events
https://api.github.com/users/dorooddorood606/repos
https://api.github.com/users/dorooddorood606/starred{/owner}{/repo}
https://api.github.com/users/dorooddorood606/subscriptions
User
https://api.github.com/users/dorooddorood606
public
Hi
I need to use "unbatch" operation in tensorflow on a huggingface dataset, I could not find this operation, could you kindly direct me how I can do it, here is the problem I am trying to solve:

I am considering "record" dataset in SuperGlue and I need to replicate each entery of the dataset for each answer, to make it similar to what T5 originally did:

https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L925

Here please find an example:

  For example, a typical example from ReCoRD might look like
  {
      'passsage': 'This is the passage.',
      'query': 'A @placeholder is a bird.',
      'entities': ['penguin', 'potato', 'pigeon'],
      'answers': ['penguin', 'pigeon'],
  }
  and I need a prosessor which would turn this example into the following two examples:
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'penguin',
  }
  and
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'pigeon',
  }


For doing this, one need unbatch, as each entry can map to multiple samples depending on the number of answers, I am not sure how to perform this operation with  huggingface datasets library and greatly appreciate your help

@lhoestq 

Thank you very much.
