Offset overflow while doing regex on a text column
COLLABORATOR
Hi! This looks like an Arrow bug, but it can be avoided by reducing the `writer_batch_size`.

(`ds = ds.map(get_text_caption, writer_batch_size=100)` in Colab runs without issues)

2023-05-05T15:57:41Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1536460310
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5blIYW
https://api.github.com/repos/huggingface/datasets/issues/comments/1536460310/reactions
2023-05-05T15:57:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1536460310
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko I ran into this problem with load_dataset. What should I do
2023-09-03T14:47:37Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1704325078
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5lle_W
https://api.github.com/repos/huggingface/datasets/issues/comments/1704325078/reactions
2023-09-03T14:47:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1704325078
https://avatars.githubusercontent.com/u/51043929?v=4
https://api.github.com/users/aihao2000/events{/privacy}
https://api.github.com/users/aihao2000/followers
https://api.github.com/users/aihao2000/following{/other_user}
https://api.github.com/users/aihao2000/gists{/gist_id}
https://github.com/aihao2000
aihao2000
MDQ6VXNlcjUxMDQzOTI5
https://api.github.com/users/aihao2000/orgs
https://api.github.com/users/aihao2000/received_events
https://api.github.com/users/aihao2000/repos
https://api.github.com/users/aihao2000/starred{/owner}{/repo}
https://api.github.com/users/aihao2000/subscriptions
User
https://api.github.com/users/aihao2000
public
COLLABORATOR
@AisingioroHao0  You can also pass the `writer_batch_size` parameter to `load_dataset`, e.g., `load_dataset("mnist", writer_batch_size=100)`
2023-09-04T14:26:29Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1705361235
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5lpb9T
https://api.github.com/repos/huggingface/datasets/issues/comments/1705361235/reactions
2023-09-04T14:26:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1705361235
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko How do I determine the optimal size of write_batch_size? My training is sometimes fast and sometimes slow. Is it because write_batch_size is too small? Each batch of the current dataloader should be the same size. I preprocessed the dataset using map
2023-09-18T12:30:19Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1723311428
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5mt6VE
https://api.github.com/repos/huggingface/datasets/issues/comments/1723311428/reactions
2023-09-18T12:30:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1723311428
https://avatars.githubusercontent.com/u/51043929?v=4
https://api.github.com/users/aihao2000/events{/privacy}
https://api.github.com/users/aihao2000/followers
https://api.github.com/users/aihao2000/following{/other_user}
https://api.github.com/users/aihao2000/gists{/gist_id}
https://github.com/aihao2000
aihao2000
MDQ6VXNlcjUxMDQzOTI5
https://api.github.com/users/aihao2000/orgs
https://api.github.com/users/aihao2000/received_events
https://api.github.com/users/aihao2000/repos
https://api.github.com/users/aihao2000/starred{/owner}{/repo}
https://api.github.com/users/aihao2000/subscriptions
User
https://api.github.com/users/aihao2000
public
COLLABORATOR
@aihao2000 It's unlikely `writer_batch_size` is the problem. You can use the following code to profile the training loop (e.g., on a subset of data) and find slow parts:
```python
import cProfile, pstats

with cProfile.Profile() as profiler:
    ... # training loop code

stats = pstats.Stats(profiler).sort_stats("cumtime")
stats.print_stats()
```

2023-09-21T13:16:27Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1729566528
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5nFxdA
https://api.github.com/repos/huggingface/datasets/issues/comments/1729566528/reactions
2023-09-21T13:16:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1729566528
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@nishanthcgit ok，thanks.Recently I found dataset.with_transform to be faster and more stable with multiple processes
2023-09-21T13:33:49Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1729598365
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5nF5Od
https://api.github.com/repos/huggingface/datasets/issues/comments/1729598365/reactions
2023-09-21T13:33:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1729598365
https://avatars.githubusercontent.com/u/51043929?v=4
https://api.github.com/users/aihao2000/events{/privacy}
https://api.github.com/users/aihao2000/followers
https://api.github.com/users/aihao2000/following{/other_user}
https://api.github.com/users/aihao2000/gists{/gist_id}
https://github.com/aihao2000
aihao2000
MDQ6VXNlcjUxMDQzOTI5
https://api.github.com/users/aihao2000/orgs
https://api.github.com/users/aihao2000/received_events
https://api.github.com/users/aihao2000/repos
https://api.github.com/users/aihao2000/starred{/owner}{/repo}
https://api.github.com/users/aihao2000/subscriptions
User
https://api.github.com/users/aihao2000
public
NONE
@mariosasko Is the larger the num_proc of load_dataset within the number of cpu cores, the better? Then the num_proc of data_loader is the number of cpu cores/number of training processes
2023-09-22T06:44:07Z
https://github.com/huggingface/datasets/issues/5783#issuecomment-1730887594
https://api.github.com/repos/huggingface/datasets/issues/5783
IC_kwDODunzps5nKz-q
https://api.github.com/repos/huggingface/datasets/issues/comments/1730887594/reactions
2023-09-22T06:44:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1730887594
https://avatars.githubusercontent.com/u/51043929?v=4
https://api.github.com/users/aihao2000/events{/privacy}
https://api.github.com/users/aihao2000/followers
https://api.github.com/users/aihao2000/following{/other_user}
https://api.github.com/users/aihao2000/gists{/gist_id}
https://github.com/aihao2000
aihao2000
MDQ6VXNlcjUxMDQzOTI5
https://api.github.com/users/aihao2000/orgs
https://api.github.com/users/aihao2000/received_events
https://api.github.com/users/aihao2000/repos
https://api.github.com/users/aihao2000/starred{/owner}{/repo}
https://api.github.com/users/aihao2000/subscriptions
User
https://api.github.com/users/aihao2000
public
### Describe the bug

`ArrowInvalid: offset overflow while concatenating arrays`

Same error as [here](https://github.com/huggingface/datasets/issues/615)

### Steps to reproduce the bug


Steps to reproduce: (dataset is a few GB big so try in colab maybe)

```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.

### Expected behavior

Dataset should have a new column with processed text

### Environment info

Datasets version - 2.11.0