extremely slow processing when using a custom dataset 
MEMBER
Hi !

My guess is that some examples in your dataset are bigger than your RAM, and therefore loading them in RAM to pass them to `remove_non_indic_sentences` takes forever because it might use SWAP memory.

Maybe several examples in your dataset are grouped together, can you check `len(lang_dataset["train"])` and `lang_dataset["train"].data.nbytes` of both datasets please ? It can also be helpful to check the distribution of lengths of each examples in your dataset.
2022-06-06T16:14:09Z
https://github.com/huggingface/datasets/issues/4374#issuecomment-1147627618
https://api.github.com/repos/huggingface/datasets/issues/4374
IC_kwDODunzps5EZ2hi
https://api.github.com/repos/huggingface/datasets/issues/comments/1147627618/reactions
2022-06-06T16:14:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1147627618
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Closing due to inactivity
2023-07-25T15:07:16Z
https://github.com/huggingface/datasets/issues/4374#issuecomment-1650023725
https://api.github.com/repos/huggingface/datasets/issues/4374
IC_kwDODunzps5iWV0t
https://api.github.com/repos/huggingface/datasets/issues/comments/1650023725/reactions
2023-07-25T15:07:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1650023725
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
## processing a custom dataset loaded as .txt file  is extremely slow,  compared to a dataset of similar volume from the hub

I have a large .txt file of 22 GB which i load into HF dataset 

`lang_dataset = datasets.load_dataset("text", data_files="hi.txt")`

further i use a pre-processing function to clean the dataset 

 `lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)`

the following processing takes astronomical time to process, while hoging all the ram. 

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data. 
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi",  use_auth_token=True)`

the hours predicted to preprocess are as follows:

huggingface hub dataset:  6.5 hrs 
custom loaded dataset: 7000 hrs

note: both the datasets are almost actually same, just provided by different sources with has +/- some samples, only one is hosted on the HF hub and the other is downloaded in a text format. 

## Steps to reproduce the bug
```
import datasets
import psutil
import sys 
import glob 
from fastcore.utils import listify
import re 
import gc 

def remove_non_indic_sentences(example): 
    tmp_ls = []
    eng_regex = r'[. a-zA-Z0-9ÖÄÅöäå _.,!"\'\/$]*'
    for e in listify(example['text']):
        matches = re.findall(eng_regex, e)
        for match in (str(match).strip() for match in matches if match not in [""," ", "  ", ",", " ,", ", ", " , "]):
            if len(list(match.split(" "))) > 2:
                e = re.sub(match," ",e,count=1)
        tmp_ls.append(e)
        gc.collect()
    example['clean_text'] = tmp_ls
    return example

lang_dataset = datasets.load_dataset("text", data_files="hi.txt")

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)


## same thing work much faster when loading similar dataset from hub
 
lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", split="train", use_auth_token=True)

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

```

## Actual results

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data. 
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", use_auth_token=True)


**the hours predicted to preprocess are as follows:**
huggingface hub dataset:  6.5 hrs 
custom loaded dataset: 7000 hrs

**i even tried the following:**

-  sharding the large 22gb text files into smaller files and loading
- saving the file to disk and then loading 
- using lesser num_proc 
- using smaller batch size 
- processing without batches ie :  without `batched=True`


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2.dev0
- Platform: Ubuntu 20.04 LTS 
- Python version: 3.9.7 
- PyArrow version:8.0.0 

