keep_in_memory=True does not seem to work
NONE
Seems like I just used the old code which did not have `keep_in_memory=True` argument, sorry.

Although i encountered a different problem – at 97% my python process just hung for around 11 minutes with no logs (when running dataset.map without `keep_in_memory=True` over around 3 million of dataset samples)...
2024-01-08T11:23:19Z
https://github.com/huggingface/datasets/issues/6568#issuecomment-1880817794
https://api.github.com/repos/huggingface/datasets/issues/6568
IC_kwDODunzps5wGwCC
https://api.github.com/repos/huggingface/datasets/issues/comments/1880817794/reactions
2024-01-08T11:26:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1880817794
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
MEMBER
Can you open a new issue and provide a bit more details ? What kind of map operations did you run ?
2024-01-09T15:23:02Z
https://github.com/huggingface/datasets/issues/6568#issuecomment-1883254114
https://api.github.com/repos/huggingface/datasets/issues/6568
IC_kwDODunzps5wQC1i
https://api.github.com/repos/huggingface/datasets/issues/comments/1883254114/reactions
2024-01-09T15:23:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1883254114
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hey. I will try to find some free time to describe it.

(can't do it now, cause i need to reproduce it myself to be sure about everything, which requires spinning a new Azuree VM, copying a huge dataset to drive from network disk for a long time etc...)
2024-01-09T19:07:43Z
https://github.com/huggingface/datasets/issues/6568#issuecomment-1883625427
https://api.github.com/repos/huggingface/datasets/issues/6568
IC_kwDODunzps5wRdfT
https://api.github.com/repos/huggingface/datasets/issues/comments/1883625427/reactions
2024-01-09T19:07:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1883625427
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
NONE
@lhoestq loading dataset like this does not spawn 50 python processes:

```
datasets.load_dataset("/preprocessed_2256k/train", num_proc=50)
```

I have 64 vCPU so i hoped it could speed up the dataset loading...

My dataset onlly has images and metadata.csv with text column alongside image file path column
2024-01-13T04:18:54Z
https://github.com/huggingface/datasets/issues/6568#issuecomment-1890299865
https://api.github.com/repos/huggingface/datasets/issues/6568
IC_kwDODunzps5wq6_Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1890299865/reactions
2024-01-13T04:18:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1890299865
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
NONE
now noticed
```
'Setting num_proc from 50 back to 1 for the train split to disable multiprocessing as it only contains one shard
```

Any way to work around this?
2024-01-13T04:31:07Z
https://github.com/huggingface/datasets/issues/6568#issuecomment-1890302893
https://api.github.com/repos/huggingface/datasets/issues/6568
IC_kwDODunzps5wq7ut
https://api.github.com/repos/huggingface/datasets/issues/comments/1890302893/reactions
2024-01-13T04:31:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1890302893
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
NONE
@lhoestq thanks, [this helped](https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/arrow_dataset.py#L1053)


2024-01-13T04:53:04Z
https://github.com/huggingface/datasets/issues/6568#issuecomment-1890308374
https://api.github.com/repos/huggingface/datasets/issues/6568
IC_kwDODunzps5wq9EW
https://api.github.com/repos/huggingface/datasets/issues/comments/1890308374/reactions
2024-01-13T04:53:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1890308374
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
UPD: [Fixed](https://github.com/huggingface/datasets/issues/6568#issuecomment-1880817794) . But a new issue came up :(