[Question] How to move and reuse preprocessed dataset? 
NONE
@lhoestq @LysandreJik
2021-05-11T09:10:47Z
https://github.com/huggingface/datasets/issues/2345#issuecomment-838125844
https://api.github.com/repos/huggingface/datasets/issues/2345
MDEyOklzc3VlQ29tbWVudDgzODEyNTg0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/838125844/reactions
2021-05-11T09:10:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/838125844
https://avatars.githubusercontent.com/u/15045402?v=4
https://api.github.com/users/AtmaHou/events{/privacy}
https://api.github.com/users/AtmaHou/followers
https://api.github.com/users/AtmaHou/following{/other_user}
https://api.github.com/users/AtmaHou/gists{/gist_id}
https://github.com/AtmaHou
AtmaHou
MDQ6VXNlcjE1MDQ1NDAy
https://api.github.com/users/AtmaHou/orgs
https://api.github.com/users/AtmaHou/received_events
https://api.github.com/users/AtmaHou/repos
https://api.github.com/users/AtmaHou/starred{/owner}{/repo}
https://api.github.com/users/AtmaHou/subscriptions
User
https://api.github.com/users/AtmaHou
public
MEMBER
<s>Hi :) Can you share with us the code you used ?</s>

EDIT: from https://github.com/huggingface/transformers/issues/11665#issuecomment-838348291 I understand you're using the run_clm.py script. Can you share your logs ?

2021-05-12T13:24:14Z
https://github.com/huggingface/datasets/issues/2345#issuecomment-839769978
https://api.github.com/repos/huggingface/datasets/issues/2345
MDEyOklzc3VlQ29tbWVudDgzOTc2OTk3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/839769978/reactions
2021-05-12T13:26:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/839769978
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same
2021-05-12T13:27:21Z
https://github.com/huggingface/datasets/issues/2345#issuecomment-839772063
https://api.github.com/repos/huggingface/datasets/issues/2345
MDEyOklzc3VlQ29tbWVudDgzOTc3MjA2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/839772063/reactions
2021-05-12T13:27:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/839772063
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

I only changed the `preprocessing_num_workers` maybe it is the problem~ I will try again~
2021-05-13T03:52:32Z
https://github.com/huggingface/datasets/issues/2345#issuecomment-840274059
https://api.github.com/repos/huggingface/datasets/issues/2345
MDEyOklzc3VlQ29tbWVudDg0MDI3NDA1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/840274059/reactions
2021-05-13T03:52:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/840274059
https://avatars.githubusercontent.com/u/15045402?v=4
https://api.github.com/users/AtmaHou/events{/privacy}
https://api.github.com/users/AtmaHou/followers
https://api.github.com/users/AtmaHou/following{/other_user}
https://api.github.com/users/AtmaHou/gists{/gist_id}
https://github.com/AtmaHou
AtmaHou
MDQ6VXNlcjE1MDQ1NDAy
https://api.github.com/users/AtmaHou/orgs
https://api.github.com/users/AtmaHou/received_events
https://api.github.com/users/AtmaHou/repos
https://api.github.com/users/AtmaHou/starred{/owner}{/repo}
https://api.github.com/users/AtmaHou/subscriptions
User
https://api.github.com/users/AtmaHou
public
Hi, I am training a gpt-2 from scratch using run_clm.py.

I want to move and reuse the preprocessed dataset (It take 2 hour to preprocess),

I tried to :

copy path_to_cache_dir/datasets to new_cache_dir/datasets
set export HF_DATASETS_CACHE="new_cache_dir/"
but the program still re-preprocess the whole dataset without loading cache.

I also tried to torch.save(lm_datasets, fw), but the saved file is only 14M.

What is the proper way to do this?