pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500
MEMBER
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```
2021-02-05T14:29:20Z
https://github.com/huggingface/datasets/issues/1817#issuecomment-774066254
https://api.github.com/repos/huggingface/datasets/issues/1817
MDEyOklzc3VlQ29tbWVudDc3NDA2NjI1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/774066254/reactions
2021-02-05T14:29:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774066254
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
You saved my life.
2021-02-07T14:04:24Z
https://github.com/huggingface/datasets/issues/1817#issuecomment-774679650
https://api.github.com/repos/huggingface/datasets/issues/1817
MDEyOklzc3VlQ29tbWVudDc3NDY3OTY1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/774679650/reactions
2021-02-07T14:04:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774679650
https://avatars.githubusercontent.com/u/9610770?v=4
https://api.github.com/users/LuCeHe/events{/privacy}
https://api.github.com/users/LuCeHe/followers
https://api.github.com/users/LuCeHe/following{/other_user}
https://api.github.com/users/LuCeHe/gists{/gist_id}
https://github.com/LuCeHe
LuCeHe
MDQ6VXNlcjk2MTA3NzA=
https://api.github.com/users/LuCeHe/orgs
https://api.github.com/users/LuCeHe/received_events
https://api.github.com/users/LuCeHe/repos
https://api.github.com/users/LuCeHe/starred{/owner}{/repo}
https://api.github.com/users/LuCeHe/subscriptions
User
https://api.github.com/users/LuCeHe
public
I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!