is there a way to override a dataset object saved with save_to_disk?
MEMBER
Hi
You can rename the arrow file and update the name in `state.json`
2021-03-16T16:30:19Z
https://github.com/huggingface/datasets/issues/2055#issuecomment-800413859
https://api.github.com/repos/huggingface/datasets/issues/2055
MDEyOklzc3VlQ29tbWVudDgwMDQxMzg1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800413859/reactions
2021-03-16T16:30:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800413859
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I tried this way, but when there is a mapping process to the dataset, it again uses a random cache name. atm, I am trying to use the following method by setting an exact cache file,

```
            dataset_with_embedding =csv_dataset.map(
                partial(self.embed, ctx_encoder=ctx_encoder, ctx_tokenizer=self.context_tokenizer),
                batched=True,
                batch_size=1,
                features=new_features,
                cache_file_name=cache_arrow_path,
                load_from_cache_file=False
            )
```
So here we set a cache_file_name , after this it uses the same  file name when saving again and again. 
2021-03-16T21:07:26Z
https://github.com/huggingface/datasets/issues/2055#issuecomment-800605325
https://api.github.com/repos/huggingface/datasets/issues/2055
MDEyOklzc3VlQ29tbWVudDgwMDYwNTMyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800605325/reactions
2021-03-16T21:07:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800605325
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
I'm not sure I understand your issue, can you elaborate ?

`cache_file_name` is indeed an argument you can set to specify the cache file that will be used for the processed dataset. By default the file is named with something like `cache-<fingerprint>.arrow` where the fingerprint is a hash.
2021-03-17T10:36:27Z
https://github.com/huggingface/datasets/issues/2055#issuecomment-800976199
https://api.github.com/repos/huggingface/datasets/issues/2055
MDEyOklzc3VlQ29tbWVudDgwMDk3NjE5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800976199/reactions
2021-03-17T10:36:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800976199
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Let's say I am updating a set of embedding in a dataset that is around 40GB inside a training loop every 500 steps (Ex: calculating the embeddings in updated ctx_encoder in RAG and saving it to the passage path).  So when we use **dataset_object.save_to_disk('passage_path_directory')** it will save the new dataset object every time with a random file name, especially when we do some transformations to dataset objects such as map or shards. This way, we keep collecting unwanted files that will eventually eat up all the disk space. 

But if we can save the dataset object every time by a single name like **data_shard_1.arrow**, it will automatically remove the previous file and save the new one in the same directory.  I found the above-mentioned code snippet useful to complete this task. 

Is this clear?
2021-03-17T10:47:15Z
https://github.com/huggingface/datasets/issues/2055#issuecomment-800982767
https://api.github.com/repos/huggingface/datasets/issues/2055
MDEyOklzc3VlQ29tbWVudDgwMDk4Mjc2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/800982767/reactions
2021-03-17T10:47:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800982767
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
At the moment when I use save_to_disk, it uses the arbitrary name for the arrow file.  Is there a way to override such an object? 