save_to_disk doesn't work when we use concatenate_datasets function before creating the final dataset_object.
MEMBER
Hi ! We refactored save_to_disk in #2025 so this doesn't happen.
Feel free to try it on master for now
We'll do a new release soon
2021-04-14T13:57:04Z
https://github.com/huggingface/datasets/issues/2189#issuecomment-819538814
https://api.github.com/repos/huggingface/datasets/issues/2189
MDEyOklzc3VlQ29tbWVudDgxOTUzODgxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/819538814/reactions
2021-04-14T13:57:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/819538814
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
As you can see, it saves the entire dataset.

@lhoestq 

You can  check by going through the following example,

```
from datasets import load_from_disk,concatenate_datasets

loaded_data=load_from_disk('/home/gsir059/HNSW-ori/my_knowledge_dataset')
n=20
kb_list=[loaded_data.shard(n, i, contiguous=True) for i in range(n)]
final_dataset=concatenate_datasets([kb_list[1],kb_list[2]])
final_dataset.save_to_disk('/home/gsir059/haha/k.arrow')
```