Formatting overhead
NONE
Ah I think the `line-profiler` log is off-by-one and it is in fact the `extract_batch` method that's taking forever. Will investigate further.
2023-11-06T22:57:19Z
https://github.com/huggingface/datasets/issues/6386#issuecomment-1796995041
https://api.github.com/repos/huggingface/datasets/issues/6386
IC_kwDODunzps5rG_fh
https://api.github.com/repos/huggingface/datasets/issues/comments/1796995041/reactions
2023-11-06T22:57:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1796995041
https://avatars.githubusercontent.com/u/320321?v=4
https://api.github.com/users/d-miketa/events{/privacy}
https://api.github.com/users/d-miketa/followers
https://api.github.com/users/d-miketa/following{/other_user}
https://api.github.com/users/d-miketa/gists{/gist_id}
https://github.com/d-miketa
d-miketa
MDQ6VXNlcjMyMDMyMQ==
https://api.github.com/users/d-miketa/orgs
https://api.github.com/users/d-miketa/received_events
https://api.github.com/users/d-miketa/repos
https://api.github.com/users/d-miketa/starred{/owner}{/repo}
https://api.github.com/users/d-miketa/subscriptions
User
https://api.github.com/users/d-miketa
public
NONE
I tracked it down to a quirk of my setup. Apologies.
2023-11-06T23:56:12Z
https://github.com/huggingface/datasets/issues/6386#issuecomment-1797050980
https://api.github.com/repos/huggingface/datasets/issues/6386
IC_kwDODunzps5rHNJk
https://api.github.com/repos/huggingface/datasets/issues/comments/1797050980/reactions
2023-11-06T23:56:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1797050980
https://avatars.githubusercontent.com/u/320321?v=4
https://api.github.com/users/d-miketa/events{/privacy}
https://api.github.com/users/d-miketa/followers
https://api.github.com/users/d-miketa/following{/other_user}
https://api.github.com/users/d-miketa/gists{/gist_id}
https://github.com/d-miketa
d-miketa
MDQ6VXNlcjMyMDMyMQ==
https://api.github.com/users/d-miketa/orgs
https://api.github.com/users/d-miketa/received_events
https://api.github.com/users/d-miketa/repos
https://api.github.com/users/d-miketa/starred{/owner}{/repo}
https://api.github.com/users/d-miketa/subscriptions
User
https://api.github.com/users/d-miketa
public
### Describe the bug

Hi! I very recently noticed that my training time is dominated by batch formatting. Using Lightning's profilers, I located the bottleneck within `datasets.formatting.formatting` and then narrowed it down with `line-profiler`. It turns out that almost all of the overhead is due to creating new instances of `self.python_arrow_extractor`. I admit I'm confused why that could be the case - as far as I can tell there's no complex `__init__` logic to execute.

![image](https://github.com/huggingface/datasets/assets/320321/5e022e0b-0d21-43d0-8e6f-9e641142e96b)

### Steps to reproduce the bug

1. Set up a dataset `ds` with potentially several (4+) columns (not sure if this is necessary, but it did at one point of the investigation make overhead worse)
2. Process it using a custom transform, `ds = ds.with_transform(transform_func)`
3. Decorate this function https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L512 with `@profile` from https://pypi.org/project/line-profiler/
4. Profile with `$ kernprof -l script_to_profile.py`

### Expected behavior

Batch formatting should have acceptable overhead.

### Environment info

```
datasets=2.14.6
pyarrow=14.0.0
```