Memory usage consistently increases when processing a dataset with `.map`
MEMBER
Hi ! Can you share the function you pass to `map` ?
I know you mentioned it would be hard to share some code but this would really help to understand what happened
2021-06-29T08:43:00Z
https://github.com/huggingface/datasets/issues/2559#issuecomment-870401083
https://api.github.com/repos/huggingface/datasets/issues/2559
MDEyOklzc3VlQ29tbWVudDg3MDQwMTA4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/870401083/reactions
2021-06-29T08:43:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/870401083
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
This is the same behavior as in #4883, so I'm closing this issue as a duplicate. 
2023-07-20T13:34:09Z
https://github.com/huggingface/datasets/issues/2559#issuecomment-1643937538
https://api.github.com/repos/huggingface/datasets/issues/2559
IC_kwDODunzps5h_H8C
https://api.github.com/repos/huggingface/datasets/issues/comments/1643937538/reactions
2023-07-20T13:34:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1643937538
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
## Describe the bug

I have a HF dataset with image paths stored in it and I am trying to load those image paths using `.map` with `num_proc=80`. I am noticing that the memory usage consistently keeps on increasing with time. I tried using `DEFAULT_WRITER_BATCH_SIZE=10` in the builder to decrease arrow writer's batch size but that doesn't seem to help.

## Steps to reproduce the bug

Providing code as it is would be hard. I can provide a MVP if that helps.

## Expected results

Memory usage should become consistent after some time following the launch of processing.

## Actual results

Memory usage keeps on increasing.

## Environment info

- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0