ds.map(f, num_proc=10) is slower than df.apply
MEMBER
Hi ! `map()` reads all the columns and writes the resulting dataset with all the columns as well, while df.column_name.apply only reads and writes one column and does it in memory. So this is speed difference is actually expected.

Moreover using multiprocessing on a dataset that lives in memory (from_pandas uses the same in-memory data as the pandas DataFrame while load_dataset or from_generator load from disk) requires to copy the data to each subprocess which can also be slow. Data loaded from disk don't need to be copied though since they work as a form of shared memory thanks to memory mapping.

However you can make you map() call much faster by making it read and write only the column you want:

```python
has_cover_ds = ds.map(lambda song_name: {'has_cover': has_cover(song_name)}, input_columns=["song_name"], remove_columns=ds.column_names)  # outputs a dataset with 1 column
ds = ds.concatenate_datasets([ds, has_cover_ds], axis=1)
```

and if your dataset is loaded from disk you can pass num_proc=10 and get a nice speed up as well (no need to copy the data to subprocesses)
2024-10-22T16:04:18Z
https://github.com/huggingface/datasets/issues/7217#issuecomment-2429685372
https://api.github.com/repos/huggingface/datasets/issues/7217
IC_kwDODunzps6Q0g58
https://api.github.com/repos/huggingface/datasets/issues/comments/2429685372/reactions
2024-10-22T16:05:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2429685372
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Isn't there a way to do memory mapping with the in-memory dataset without saving it to disk?
2025-02-28T21:20:00Z
https://github.com/huggingface/datasets/issues/7217#issuecomment-2691564647
https://api.github.com/repos/huggingface/datasets/issues/7217
IC_kwDODunzps6gbgRn
https://api.github.com/repos/huggingface/datasets/issues/comments/2691564647/reactions
2025-02-28T21:20:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2691564647
https://avatars.githubusercontent.com/u/3905501?v=4
https://api.github.com/users/bryant1410/events{/privacy}
https://api.github.com/users/bryant1410/followers
https://api.github.com/users/bryant1410/following{/other_user}
https://api.github.com/users/bryant1410/gists{/gist_id}
https://github.com/bryant1410
bryant1410
MDQ6VXNlcjM5MDU1MDE=
https://api.github.com/users/bryant1410/orgs
https://api.github.com/users/bryant1410/received_events
https://api.github.com/users/bryant1410/repos
https://api.github.com/users/bryant1410/starred{/owner}{/repo}
https://api.github.com/users/bryant1410/subscriptions
User
https://api.github.com/users/bryant1410
public
CONTRIBUTOR
Maybe saving it to a memory-mapped filesystem? It'd be like a trick to make datasets save to "disk" but actually it's memory. But it feels like there should be a better "automatic" way provided by `datasets`.
2025-02-28T21:21:01Z
https://github.com/huggingface/datasets/issues/7217#issuecomment-2691565995
https://api.github.com/repos/huggingface/datasets/issues/7217
IC_kwDODunzps6gbgmr
https://api.github.com/repos/huggingface/datasets/issues/comments/2691565995/reactions
2025-02-28T21:21:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2691565995
https://avatars.githubusercontent.com/u/3905501?v=4
https://api.github.com/users/bryant1410/events{/privacy}
https://api.github.com/users/bryant1410/followers
https://api.github.com/users/bryant1410/following{/other_user}
https://api.github.com/users/bryant1410/gists{/gist_id}
https://github.com/bryant1410
bryant1410
MDQ6VXNlcjM5MDU1MDE=
https://api.github.com/users/bryant1410/orgs
https://api.github.com/users/bryant1410/received_events
https://api.github.com/users/bryant1410/repos
https://api.github.com/users/bryant1410/starred{/owner}{/repo}
https://api.github.com/users/bryant1410/subscriptions
User
https://api.github.com/users/bryant1410
public
### Describe the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

time cost: 
1. df.apply: 100%|██████████| 12500592/12500592 [00:13<00:00, 959825.47it/s]
2. ds.map: Map (num_proc=10):  31%
 3899028/12500592 [00:28<00:38, 222532.89 examples/s]

### Steps to reproduce the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

### Expected behavior

ds.map is ~num_proc faster than df.apply

### Environment info

pandas: 2.2.2
datasets: 2.19.1