How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 
MEMBER
Hi ! That looks like a bug, can you provide some code so that we can reproduce ?
It's not supposed to update the original dataset
2021-03-09T11:05:44Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-793723926
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5MzcyMzkyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/793723926/reactions
2021-03-09T11:05:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/793723926
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, I experimented with RAG. 

Actually, you can run the [use_own_knowldge_dataset.py](https://github.com/shamanez/transformers/blob/rag-end-to-end-retrieval/examples/research_projects/rag/use_own_knowledge_dataset.py#L80). In the 80 you can save the dataset object to the disk with save_to_disk. Then in order to compute the embeddings in this use **load_from_disk**. 

Then finally save it. You can see the original dataset object (CSV after splitting also will be changed)

One more thing- when I save the dataset object with **save_to_disk** it name the arrow file with cache.... rather than using dataset. arrow.  Can you add a variable that we can feed a name to save_to_disk function?
2021-03-09T11:57:16Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-793780885
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5Mzc4MDg4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/793780885/reactions
2021-03-09T11:57:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/793780885
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
NONE
@lhoestq   I also found that cache in tmp directory gets updated after transformations. This is really problematic when using datasets interactively. Let's say we use the shards function to a dataset loaded with csv, atm when we do transformations to shards and combine them it updates the original csv cache. 
2021-03-11T23:26:13Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-797118118
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5NzExODExOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/797118118/reactions
2021-03-11T23:26:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797118118
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
I plan to update the save_to_disk method in #2025 so I can make sure the new save_to_disk doesn't corrupt your cache files.
But from your last message it looks like save_to_disk isn't the root cause right ?
2021-03-12T12:45:29Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-797467936
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5NzQ2NzkzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/797467936/reactions
2021-03-12T12:45:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797467936
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
ok, one more thing. When we use save_to_disk there are two files other than .arrow. dataset_info.json and state.json. Sometimes most of the fields in the dataset_infor.json are null, especially when saving dataset objects. Anyways I think load_from_disk uses the arrow files mentioned in state.json right? 
2021-03-12T13:00:12Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-797474938
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5NzQ3NDkzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/797474938/reactions
2021-03-12T13:00:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797474938
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
> Anyways I think load_from_disk uses the arrow files mentioned in state.json right?

Yes exactly
2021-03-12T13:28:32Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-797490293
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5NzQ5MDI5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/797490293/reactions
2021-03-12T13:28:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797490293
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Perfect.  For now, I am loading the dataset from CSV in my interactive process and will wait until you make the PR!
2021-03-12T13:31:32Z
https://github.com/huggingface/datasets/issues/1993#issuecomment-797491997
https://api.github.com/repos/huggingface/datasets/issues/1993
MDEyOklzc3VlQ29tbWVudDc5NzQ5MTk5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/797491997/reactions
2021-03-12T13:31:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797491997
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
I am using the latest datasets library.  In my work, I first use **load_from_disk** to load a data set that contains 3.8Gb information. Then during my training process, I update that dataset object and add new elements and save it in a different place.  

When I save the dataset with **save_to_disk**, the original dataset which is already in the disk also gets updated. I do not want to update it.  How to prevent from this?
