Parquet uploads off-by-one naming scheme
COLLABORATOR
You can find the reasoning behind this naming scheme [here](https://github.com/huggingface/transformers/pull/16343#discussion_r931182168).

This point has been raised several times, so I'd be okay with starting with `00001-` (also to be consistent with the `transformers` sharding), but I'm not sure @lhoestq agrees.
2023-10-16T13:11:50Z
https://github.com/huggingface/datasets/issues/6303#issuecomment-1764457528
https://api.github.com/repos/huggingface/datasets/issues/6303
IC_kwDODunzps5pK3w4
https://api.github.com/repos/huggingface/datasets/issues/comments/1764457528/reactions
2023-10-16T13:11:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1764457528
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
We start at 0 in `datasets` for consistency with Apache Spark, Apache Beam, Dask and others.

Also note `transformers` isn't a good reference on this topic. I talked with the maintainers when they added shards but it was already released this way. Though we found that there is a backward-compatible way in `transformers` to start at 0, but no request from `transformers` users to changes this AFAIK.
2023-10-16T13:24:06Z
https://github.com/huggingface/datasets/issues/6303#issuecomment-1764479687
https://api.github.com/repos/huggingface/datasets/issues/6303
IC_kwDODunzps5pK9LH
https://api.github.com/repos/huggingface/datasets/issues/comments/1764479687/reactions
2023-10-16T13:24:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1764479687
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
not sure it would be a good idea to break the consistency now, IMO
2023-10-16T13:51:04Z
https://github.com/huggingface/datasets/issues/6303#issuecomment-1764529290
https://api.github.com/repos/huggingface/datasets/issues/6303
IC_kwDODunzps5pLJSK
https://api.github.com/repos/huggingface/datasets/issues/comments/1764529290/reactions
2023-10-16T13:51:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1764529290
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
CONTRIBUTOR
Makes sense to start at 0 for plenty of good reasons so I'm on board.

What about the second part `-of-0000X`? With single commit PR #6269  just getting merged, there was a note about issues with 100+ file edits https://github.com/huggingface/datasets/pull/6269#issuecomment-1755428581.

That would be my last remaining concern in the context of the `push_to_hub(..., append=True)` work to be done, where appending a single file to the full dataset will require renaming every other existing file in the dataset. If it doesn't seem like a big issue for this work then all the better 👍
2023-10-16T16:25:46Z
https://github.com/huggingface/datasets/issues/6303#issuecomment-1764845886
https://api.github.com/repos/huggingface/datasets/issues/6303
IC_kwDODunzps5pMWk-
https://api.github.com/repos/huggingface/datasets/issues/comments/1764845886/reactions
2023-10-16T16:33:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1764845886
https://avatars.githubusercontent.com/u/1981179?v=4
https://api.github.com/users/ZachNagengast/events{/privacy}
https://api.github.com/users/ZachNagengast/followers
https://api.github.com/users/ZachNagengast/following{/other_user}
https://api.github.com/users/ZachNagengast/gists{/gist_id}
https://github.com/ZachNagengast
ZachNagengast
MDQ6VXNlcjE5ODExNzk=
https://api.github.com/users/ZachNagengast/orgs
https://api.github.com/users/ZachNagengast/received_events
https://api.github.com/users/ZachNagengast/repos
https://api.github.com/users/ZachNagengast/starred{/owner}{/repo}
https://api.github.com/users/ZachNagengast/subscriptions
User
https://api.github.com/users/ZachNagengast
public
### Describe the bug

I noticed this numbering scheme not matching up in a different project and wanted to raise it as an issue for discussion, what is the actual proper way to have these stored?

<img width="425" alt="image" src="https://github.com/huggingface/datasets/assets/1981179/3ffa2144-7c9a-446f-b521-a5e9db71e7ce">

The `-SSSSS-of-NNNNN` seems to be used widely across the codebase. The section that creates the part in my screenshot is here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5287
There are also some edits to this section in the single commit branch.

### Steps to reproduce the bug

1. Upload a dataset that requires at least two parquet files in it
2. Observe the naming scheme

### Expected behavior

The couple options here are of course **1. keeping it as is**

**2. Starting the index at 1:**
train-00001-of-00002-{hash}.parquet
train-00002-of-00002-{hash}.parquet

**3. My preferred option** (which would solve my specific issue), dropping the total entirely:
train-00000-{hash}.parquet
train-00001-{hash}.parquet

This also solves an issue that will occur with an `append` variable for `push_to_hub` (see https://github.com/huggingface/datasets/issues/6290) where as you add a new parquet file, you need to rename everything in the repo as well. 

However, I know there are parts of the repo that use 0 as the starting file or may require the total, so raising the question for discussion.


### Environment info

- `datasets` version: 2.14.6.dev0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.18.0
- PyArrow version: 12.0.1
- Pandas version: 1.5.3