Implement proper checkpointing for dataset uploading with resume function that does not require remapping shards that have already been uploaded
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6056). All of your documentation changes will be reflected on that endpoint.
2023-07-21T13:27:22Z
https://github.com/huggingface/datasets/pull/6056#issuecomment-1645583830
https://api.github.com/repos/huggingface/datasets/issues/6056
IC_kwDODunzps5iFZ3W
https://api.github.com/repos/huggingface/datasets/issues/comments/1645583830/reactions
2023-07-21T13:27:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1645583830
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
@lhoestq Reading the filenames is something I tried earlier, but I decided to use the yaml direction because:

1. The yaml file name is constructed to retain information about the shard_size, and total number of shards, hence ensuring that the files uploaded are not just files that have the same name but actually represent a different configuration of shard_size, and total number of shards. 
2. Remembering the total file size is done easily in the yaml, whereas alternatively I am not sure how one could access the file size of the uploaded files without downloading them.
3. I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it. 

If 1 and 2 can be achieved without an additional yaml, then I would be willing to make those changes. Let me know of any ideas. 1. could be done by changing the data file names, but I'd rather not do that as to prevent breaking existing datasets that try to upload updates to their data. 
2023-07-24T14:52:53Z
https://github.com/huggingface/datasets/pull/6056#issuecomment-1648067889
https://api.github.com/repos/huggingface/datasets/issues/6056
IC_kwDODunzps5iO4Ux
https://api.github.com/repos/huggingface/datasets/issues/comments/1648067889/reactions
2023-07-24T14:52:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1648067889
https://avatars.githubusercontent.com/u/10792502?v=4
https://api.github.com/users/AntreasAntoniou/events{/privacy}
https://api.github.com/users/AntreasAntoniou/followers
https://api.github.com/users/AntreasAntoniou/following{/other_user}
https://api.github.com/users/AntreasAntoniou/gists{/gist_id}
https://github.com/AntreasAntoniou
AntreasAntoniou
MDQ6VXNlcjEwNzkyNTAy
https://api.github.com/users/AntreasAntoniou/orgs
https://api.github.com/users/AntreasAntoniou/received_events
https://api.github.com/users/AntreasAntoniou/repos
https://api.github.com/users/AntreasAntoniou/starred{/owner}{/repo}
https://api.github.com/users/AntreasAntoniou/subscriptions
User
https://api.github.com/users/AntreasAntoniou
public
MEMBER
If the file name depends on the shard's fingerprint **before** mapping then we can know if a shard has been uploaded before mapping and without requiring an extra YAML file. It should do the job imo

> I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it.

what was the issue ?
2023-07-24T15:17:28Z
https://github.com/huggingface/datasets/pull/6056#issuecomment-1648116489
https://api.github.com/repos/huggingface/datasets/issues/6056
IC_kwDODunzps5iPEMJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1648116489/reactions
2023-07-24T15:17:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1648116489
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> If the file name depends on the shard's fingerprint **before** mapping then we can know if a shard has been uploaded before mapping and without requiring an extra YAML file. It should do the job imo
> 
> > I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it.
> 
> what was the issue ?

You are right. I was having some other issue earlier that I need more input from you guys to overcome, and when I overcame it the yaml was just legacy from before. I'll update the PR. 
2023-08-16T23:22:03Z
https://github.com/huggingface/datasets/pull/6056#issuecomment-1681390151
https://api.github.com/repos/huggingface/datasets/issues/6056
IC_kwDODunzps5kN_pH
https://api.github.com/repos/huggingface/datasets/issues/comments/1681390151/reactions
2023-08-16T23:22:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681390151
https://avatars.githubusercontent.com/u/10792502?v=4
https://api.github.com/users/AntreasAntoniou/events{/privacy}
https://api.github.com/users/AntreasAntoniou/followers
https://api.github.com/users/AntreasAntoniou/following{/other_user}
https://api.github.com/users/AntreasAntoniou/gists{/gist_id}
https://github.com/AntreasAntoniou
AntreasAntoniou
MDQ6VXNlcjEwNzkyNTAy
https://api.github.com/users/AntreasAntoniou/orgs
https://api.github.com/users/AntreasAntoniou/received_events
https://api.github.com/users/AntreasAntoniou/repos
https://api.github.com/users/AntreasAntoniou/starred{/owner}{/repo}
https://api.github.com/users/AntreasAntoniou/subscriptions
User
https://api.github.com/users/AntreasAntoniou
public
NONE
> If the file name depends on the shard's fingerprint **before** mapping then we can know if a shard has been uploaded before mapping and without requiring an extra YAML file. It should do the job imo
> 
> > I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it.
> 
> what was the issue ?

I remembered what it was, and why I needed the yaml. I needed it so it could remember the progress for a particular num_shards setup, as different num_shards would lead to different number of splits, and a user might switch between them while uploading, and I did not want the index to be conflated with one of another num_shards setup. 

Any idea how we deal with that without a yaml?
2023-08-16T23:39:46Z
https://github.com/huggingface/datasets/pull/6056#issuecomment-1681400959
https://api.github.com/repos/huggingface/datasets/issues/6056
IC_kwDODunzps5kOCR_
https://api.github.com/repos/huggingface/datasets/issues/comments/1681400959/reactions
2023-08-16T23:39:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681400959
https://avatars.githubusercontent.com/u/10792502?v=4
https://api.github.com/users/AntreasAntoniou/events{/privacy}
https://api.github.com/users/AntreasAntoniou/followers
https://api.github.com/users/AntreasAntoniou/following{/other_user}
https://api.github.com/users/AntreasAntoniou/gists{/gist_id}
https://github.com/AntreasAntoniou
AntreasAntoniou
MDQ6VXNlcjEwNzkyNTAy
https://api.github.com/users/AntreasAntoniou/orgs
https://api.github.com/users/AntreasAntoniou/received_events
https://api.github.com/users/AntreasAntoniou/repos
https://api.github.com/users/AntreasAntoniou/starred{/owner}{/repo}
https://api.github.com/users/AntreasAntoniou/subscriptions
User
https://api.github.com/users/AntreasAntoniou
public
MEMBER
If the user changes the num_shards parameters then we should re-upload everything.

It happens that the num_shards is part of the parquet file names, so it restarts the upload from scratch without having to write additional logic :)
2023-08-17T08:26:53Z
https://github.com/huggingface/datasets/pull/6056#issuecomment-1681855570
https://api.github.com/repos/huggingface/datasets/issues/6056
IC_kwDODunzps5kPxRS
https://api.github.com/repos/huggingface/datasets/issues/comments/1681855570/reactions
2023-08-17T08:26:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681855570
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Context: issue #5990

In order to implement the checkpointing, I introduce a metadata folder that keeps one yaml file for each set that one is uploading. This yaml keeps track of what shards have already been uploaded, and which one the idx of the latest one was. Using this information I am then able to easily get the push_to_hub function to retrieve on demand past history of uploads and continue mapping and uploading from where it was left off. 