Tokenization slows towards end of dataset
MEMBER
Hi ! First note that if the dataset is not heterogeneous / shuffled, there might be places in the data with shorter texts that are faster to tokenize.

Moreover, the way `num_proc` works is by slicing the dataset and passing each slice to a process to run the `map()` function. So at the very end of `map()`, some processes might have finished transforming their slice of data while others are still running, causing the throughput to become lower.
2024-03-15T11:34:42Z
https://github.com/huggingface/datasets/issues/6734#issuecomment-1999470313
https://api.github.com/repos/huggingface/datasets/issues/6734
IC_kwDODunzps53LX7p
https://api.github.com/repos/huggingface/datasets/issues/comments/1999470313/reactions
2024-03-15T11:34:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1999470313
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I did see some comments about how num_proc=None could help and outputting numpy arrays can also help in the docs, but this seems quite odd now dropping down to 1it/s

```bash
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46048888/46390354 [12:33:30<4:20:32, 21.84 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46049888/46390354 [12:36:11<8:37:59, 10.95 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46050888/46390354 [12:46:35<24:56:56,  3.78 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46051888/46390354 [12:56:43<35:08:10,  2.68 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46052888/46390354 [13:06:58<42:05:41,  2.23 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46053888/46390354 [13:16:01<44:40:18,  2.09 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46054888/46390354 [13:25:11<46:35:28,  2.00 examples/s]
Running tokenizer on dataset (num_proc=48):  99%|█████████▉| 46055888/46390354 [13:34:23<47:55:34,  1.94 examples/s]
```


2024-03-15T15:27:58Z
https://github.com/huggingface/datasets/issues/6734#issuecomment-1999908858
https://api.github.com/repos/huggingface/datasets/issues/6734
IC_kwDODunzps53NC_6
https://api.github.com/repos/huggingface/datasets/issues/comments/1999908858/reactions
2024-03-15T15:27:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1999908858
https://avatars.githubusercontent.com/u/98723285?v=4
https://api.github.com/users/ethansmith2000/events{/privacy}
https://api.github.com/users/ethansmith2000/followers
https://api.github.com/users/ethansmith2000/following{/other_user}
https://api.github.com/users/ethansmith2000/gists{/gist_id}
https://github.com/ethansmith2000
ethansmith2000
U_kgDOBeJl1Q
https://api.github.com/users/ethansmith2000/orgs
https://api.github.com/users/ethansmith2000/received_events
https://api.github.com/users/ethansmith2000/repos
https://api.github.com/users/ethansmith2000/starred{/owner}{/repo}
https://api.github.com/users/ethansmith2000/subscriptions
User
https://api.github.com/users/ethansmith2000
public
NONE
@ethansmith2000 Hi, did you solve this problem? I'm strugging with the same problem now.
2024-04-11T10:48:06Z
https://github.com/huggingface/datasets/issues/6734#issuecomment-2049417867
https://api.github.com/repos/huggingface/datasets/issues/6734
IC_kwDODunzps56J6KL
https://api.github.com/repos/huggingface/datasets/issues/comments/2049417867/reactions
2024-04-11T10:48:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2049417867
https://avatars.githubusercontent.com/u/28565053?v=4
https://api.github.com/users/lsh0520/events{/privacy}
https://api.github.com/users/lsh0520/followers
https://api.github.com/users/lsh0520/following{/other_user}
https://api.github.com/users/lsh0520/gists{/gist_id}
https://github.com/lsh0520
lsh0520
MDQ6VXNlcjI4NTY1MDUz
https://api.github.com/users/lsh0520/orgs
https://api.github.com/users/lsh0520/received_events
https://api.github.com/users/lsh0520/repos
https://api.github.com/users/lsh0520/starred{/owner}{/repo}
https://api.github.com/users/lsh0520/subscriptions
User
https://api.github.com/users/lsh0520
public
NONE
So, is there a way to solve this problem now?
2025-02-20T17:40:51Z
https://github.com/huggingface/datasets/issues/6734#issuecomment-2672222502
https://api.github.com/repos/huggingface/datasets/issues/6734
IC_kwDODunzps6fRuEm
https://api.github.com/repos/huggingface/datasets/issues/comments/2672222502/reactions
2025-02-20T17:40:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2672222502
https://avatars.githubusercontent.com/u/75468610?v=4
https://api.github.com/users/QZJ-2003/events{/privacy}
https://api.github.com/users/QZJ-2003/followers
https://api.github.com/users/QZJ-2003/following{/other_user}
https://api.github.com/users/QZJ-2003/gists{/gist_id}
https://github.com/QZJ-2003
QZJ-2003
MDQ6VXNlcjc1NDY4NjEw
https://api.github.com/users/QZJ-2003/orgs
https://api.github.com/users/QZJ-2003/received_events
https://api.github.com/users/QZJ-2003/repos
https://api.github.com/users/QZJ-2003/starred{/owner}{/repo}
https://api.github.com/users/QZJ-2003/subscriptions
User
https://api.github.com/users/QZJ-2003
public
### Describe the bug

Mapped tokenization slows down substantially towards end of dataset.

train set started off very slow, caught up to 20k then tapered off til the end.

what's particularly strange is that the tokenization crashed a few times before due to errors with invalid tokens somewhere or corrupted downloads, and the speed ups/downs consistently happened the same times

```bash
Running tokenizer on dataset (num_proc=48):   0%|          | 847000/881416735 [12:18<252:45:45, 967.72 examples/s]
Running tokenizer on dataset (num_proc=48):   0%|          | 848000/881416735 [12:19<224:16:10, 1090.66 examples/s]

Running tokenizer on dataset (num_proc=48):  10%|▉         | 84964000/881416735 [3:48:00<11:21:34, 19476.01 examples/s]
Running tokenizer on dataset (num_proc=48):  10%|▉         | 84967000/881416735 [3:48:00<12:04:01, 18333.79 examples/s]

Running tokenizer on dataset (num_proc=48):  61%|██████    | 538631977/881416735 [13:46:40<27:50:04, 3420.84 examples/s]
Running tokenizer on dataset (num_proc=48):  61%|██████    | 538632977/881416735 [13:46:40<23:48:20, 3999.77 examples/s]

Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881365886/881416735 [38:30:19<04:34, 185.10 examples/s]
Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881366886/881416735 [38:30:25<04:36, 180.57 examples/s]
```

and validation set as well

```bash
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41544000/46390354 [28:44<02:37, 30798.76 examples/s]
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41550000/46390354 [28:44<02:08, 37698.08 examples/s]

Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:15:48<12:22:44, 36.87 examples/s]
Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:16:00<12:22:44, 36.87 examples/s]

```



### Steps to reproduce the bug

    using the following kwargs
    
```python
with accelerator.main_process_first():
        lm_datasets = tokenized_datasets.map(
            group_texts,
            batched=True,
            num_proc=48
            load_from_cache_file=True,
            desc=f"Grouping texts in chunks of {block_size}",
        )
```

running through slurm script 
```bash
#SBATCH --partition=gpu-nvidia-a100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=96
```

using this dataset https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T





### Expected behavior

Constant speed throughout

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.10
- Python version: 3.8.18
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0