Extreme inefficiency for `save_to_disk` when merging datasets
MEMBER
Concatenating datasets doesn't create any indices mapping - so flattening indices is not needed (unless you shuffle the dataset).
Can you share the snippet of code you are using to merge your datasets and save them to disk ?
2023-12-30T15:05:46Z
https://github.com/huggingface/datasets/issues/6540#issuecomment-1872544841
https://api.github.com/repos/huggingface/datasets/issues/6540
IC_kwDODunzps5vnMRJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1872544841/reactions
2023-12-30T15:05:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1872544841
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much! 

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0 
tokenizers                0.13.1
transformers              4.31.0