Skipping shard in the remote repo and resume upload
COLLABORATOR
Hi! `_select_contiguous` fetches a (zero-copy) slice of the dataset's Arrow table to build a shard, so I don't think this part is the problem. To me, the issue seems to be the step where we embed external image files' bytes (a lot of file reads). You can use `.map` with multiprocessing to perform this step before `push_to_hub` in a faster manner and cache it to disk:
```python
from datasets.table import embed_table_storage
# load_dataset(...)
format = dataset.format
dataset = dataset.with_format("arrow")
dataset = dataset.map(embed_table_storage, batched=True)
dataset = dataset.with_format(**format)
# push_to_hub(...)
```

(In Datasets 3.0, these external bytes will be written to an Arrow file when generating a dataset to avoid this "embed" step)
2023-07-19T17:00:57Z
https://github.com/huggingface/datasets/issues/6051#issuecomment-1642443668
https://api.github.com/repos/huggingface/datasets/issues/6051
IC_kwDODunzps5h5bOU
https://api.github.com/repos/huggingface/datasets/issues/comments/1642443668/reactions
2023-07-19T17:00:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1642443668
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi, thanks, this solution saves some time.
But can't we avoid embedding all external image files bytes with each push, skipping the images that have already been pushed into the repo?

Edit: Ok I missed the part of cache it manually on the disk the first time, this solves the problem. Thank you
2023-07-20T11:54:46Z
https://github.com/huggingface/datasets/issues/6051#issuecomment-1643788539
https://api.github.com/repos/huggingface/datasets/issues/6051
IC_kwDODunzps5h-jj7
https://api.github.com/repos/huggingface/datasets/issues/comments/1643788539/reactions
2023-07-20T12:00:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1643788539
https://avatars.githubusercontent.com/u/9029817?v=4
https://api.github.com/users/rs9000/events{/privacy}
https://api.github.com/users/rs9000/followers
https://api.github.com/users/rs9000/following{/other_user}
https://api.github.com/users/rs9000/gists{/gist_id}
https://github.com/rs9000
rs9000
MDQ6VXNlcjkwMjk4MTc=
https://api.github.com/users/rs9000/orgs
https://api.github.com/users/rs9000/received_events
https://api.github.com/users/rs9000/repos
https://api.github.com/users/rs9000/starred{/owner}{/repo}
https://api.github.com/users/rs9000/subscriptions
User
https://api.github.com/users/rs9000
public
### Describe the bug

For some reason when I try to resume the upload of my dataset, it is very slow to reach the index of the shard from which to resume the uploading.

From my understanding, the problem is in this part of the code:

arrow_dataset.py
```python
for index, shard in logging.tqdm(
            enumerate(itertools.chain([first_shard], shards_iter)),
            desc="Pushing dataset shards to the dataset hub",
            total=num_shards,
            disable=not logging.is_progress_bar_enabled(),
        ):
            shard_path_in_repo = path_in_repo(index, shard)
            # Upload a shard only if it doesn't already exist in the repository
            if shard_path_in_repo not in data_files:
```

In particular, iterating the generator is slow during the call:
```python
self._select_contiguous(start, length, new_fingerprint=new_fingerprint)
```

I wonder if it is possible to avoid calling this function for shards that are already uploaded and just start from the correct shard index.

### Steps to reproduce the bug

1. Start the upload
```python
dataset = load_dataset("imagefolder", data_dir=DATA_DIR, split="train", drop_labels=True)    
dataset.push_to_hub("repo/name")
```
2. Stop and restart the upload after hundreds of shards

### Expected behavior

Skip the uploaded shards faster.

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-4.18.0-193.el8.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- PyArrow version: 12.0.1
- Pandas version: 2.0.2
