Only two cores are getting used in sagemaker with pytorch 3.10 kernel
COLLABORATOR
I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625
2023-06-26T16:17:43Z
https://github.com/huggingface/datasets/issues/5981#issuecomment-1607811182
https://api.github.com/repos/huggingface/datasets/issues/5981
IC_kwDODunzps5f1UBu
https://api.github.com/repos/huggingface/datasets/issues/comments/1607811182/reactions
2023-06-26T16:19:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1607811182
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration.
2023-06-26T19:53:47Z
https://github.com/huggingface/datasets/issues/5981#issuecomment-1608140952
https://api.github.com/repos/huggingface/datasets/issues/5981
IC_kwDODunzps5f2kiY
https://api.github.com/repos/huggingface/datasets/issues/comments/1608140952/reactions
2023-06-26T19:53:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1608140952
https://avatars.githubusercontent.com/u/107141022?v=4
https://api.github.com/users/mmr-crexi/events{/privacy}
https://api.github.com/users/mmr-crexi/followers
https://api.github.com/users/mmr-crexi/following{/other_user}
https://api.github.com/users/mmr-crexi/gists{/gist_id}
https://github.com/mmr-crexi
mmr-crexi
U_kgDOBmLXng
https://api.github.com/users/mmr-crexi/orgs
https://api.github.com/users/mmr-crexi/received_events
https://api.github.com/users/mmr-crexi/repos
https://api.github.com/users/mmr-crexi/starred{/owner}{/repo}
https://api.github.com/users/mmr-crexi/subscriptions
User
https://api.github.com/users/mmr-crexi
public
COLLABORATOR
This is too specific and unrelated to `datasets`, so this shouldn't be fixed here.
2023-07-06T16:53:33Z
https://github.com/huggingface/datasets/issues/5981#issuecomment-1624002338
https://api.github.com/repos/huggingface/datasets/issues/5981
IC_kwDODunzps5gzE8i
https://api.github.com/repos/huggingface/datasets/issues/comments/1624002338/reactions
2023-07-06T16:53:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1624002338
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko @mmr-crexi  I had the exact same problem on my kubernetes cluster. the datasets subprocess only user 1 and 17 core
2023-10-30T06:17:40Z
https://github.com/huggingface/datasets/issues/5981#issuecomment-1784548510
https://api.github.com/repos/huggingface/datasets/issues/5981
IC_kwDODunzps5qXgye
https://api.github.com/repos/huggingface/datasets/issues/comments/1784548510/reactions
2023-10-30T06:17:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1784548510
https://avatars.githubusercontent.com/u/1982475?v=4
https://api.github.com/users/nezhazheng/events{/privacy}
https://api.github.com/users/nezhazheng/followers
https://api.github.com/users/nezhazheng/following{/other_user}
https://api.github.com/users/nezhazheng/gists{/gist_id}
https://github.com/nezhazheng
nezhazheng
MDQ6VXNlcjE5ODI0NzU=
https://api.github.com/users/nezhazheng/orgs
https://api.github.com/users/nezhazheng/received_events
https://api.github.com/users/nezhazheng/repos
https://api.github.com/users/nezhazheng/starred{/owner}{/repo}
https://api.github.com/users/nezhazheng/subscriptions
User
https://api.github.com/users/nezhazheng
public
### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel