Out of memory error on workers while running Beam+Dataflow
MEMBER
Some naive ideas to cope with this:
- enable more RAM on each worker
- force the spanning of more workers
- others?
2022-06-20T07:36:05Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1160083472
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5FJXgQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1160083472/reactions
2022-06-20T07:36:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1160083472
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
@albertvillanova We were finally able to process the full NQ dataset on our machines using 600 gb with 5 workers. Maybe these numbers will work for you as well.
2022-06-23T19:46:15Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1164803073
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5FbXwB
https://api.github.com/repos/huggingface/datasets/issues/comments/1164803073/reactions
2022-06-23T19:46:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164803073
https://avatars.githubusercontent.com/u/4257308?v=4
https://api.github.com/users/seirasto/events{/privacy}
https://api.github.com/users/seirasto/followers
https://api.github.com/users/seirasto/following{/other_user}
https://api.github.com/users/seirasto/gists{/gist_id}
https://github.com/seirasto
seirasto
MDQ6VXNlcjQyNTczMDg=
https://api.github.com/users/seirasto/orgs
https://api.github.com/users/seirasto/received_events
https://api.github.com/users/seirasto/repos
https://api.github.com/users/seirasto/starred{/owner}{/repo}
https://api.github.com/users/seirasto/subscriptions
User
https://api.github.com/users/seirasto
public
MEMBER
Thanks a lot for the hint, @seirasto.

I have one question: what runner did you use? Direct, Apache Flink/Nemo/Samza/Spark, Google Dataflow...? Thank you.
2022-06-24T05:26:18Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1165209418
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5Fc69K
https://api.github.com/repos/huggingface/datasets/issues/comments/1165209418/reactions
2022-06-24T05:26:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1165209418
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
I asked my colleague who ran the code and he said apache beam.
2022-06-28T03:28:05Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1168176055
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5FoPO3
https://api.github.com/repos/huggingface/datasets/issues/comments/1168176055/reactions
2022-06-28T03:28:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1168176055
https://avatars.githubusercontent.com/u/4257308?v=4
https://api.github.com/users/seirasto/events{/privacy}
https://api.github.com/users/seirasto/followers
https://api.github.com/users/seirasto/following{/other_user}
https://api.github.com/users/seirasto/gists{/gist_id}
https://github.com/seirasto
seirasto
MDQ6VXNlcjQyNTczMDg=
https://api.github.com/users/seirasto/orgs
https://api.github.com/users/seirasto/received_events
https://api.github.com/users/seirasto/repos
https://api.github.com/users/seirasto/starred{/owner}{/repo}
https://api.github.com/users/seirasto/subscriptions
User
https://api.github.com/users/seirasto
public
CONTRIBUTOR
@albertvillanova  Since we have already processed the NQ dataset on our machines can we upload it to datasets so the NQ PR can be merged?
2022-06-28T14:25:02Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1168795650
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5FqmgC
https://api.github.com/repos/huggingface/datasets/issues/comments/1168795650/reactions
2022-06-28T14:25:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1168795650
https://avatars.githubusercontent.com/u/4257308?v=4
https://api.github.com/users/seirasto/events{/privacy}
https://api.github.com/users/seirasto/followers
https://api.github.com/users/seirasto/following{/other_user}
https://api.github.com/users/seirasto/gists{/gist_id}
https://github.com/seirasto
seirasto
MDQ6VXNlcjQyNTczMDg=
https://api.github.com/users/seirasto/orgs
https://api.github.com/users/seirasto/received_events
https://api.github.com/users/seirasto/repos
https://api.github.com/users/seirasto/starred{/owner}{/repo}
https://api.github.com/users/seirasto/subscriptions
User
https://api.github.com/users/seirasto
public
MEMBER
Maybe @lhoestq can give a more accurate answer as I am not sure about the authentication requirements to upload those files to our cloud bucket.

Anyway I propose to continue this discussion on the dedicated PR for Natural questions dataset:
- #4368
2022-06-28T14:34:14Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1168807236
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5FqpVE
https://api.github.com/repos/huggingface/datasets/issues/comments/1168807236/reactions
2022-06-28T14:37:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1168807236
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
> I asked my colleague who ran the code and he said apache beam.

He looked into it further and he just used DirectRunner. @albertvillanova 
2022-06-30T01:30:23Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1170653061
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5Fxr-F
https://api.github.com/repos/huggingface/datasets/issues/comments/1170653061/reactions
2022-06-30T01:30:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170653061
https://avatars.githubusercontent.com/u/4257308?v=4
https://api.github.com/users/seirasto/events{/privacy}
https://api.github.com/users/seirasto/followers
https://api.github.com/users/seirasto/following{/other_user}
https://api.github.com/users/seirasto/gists{/gist_id}
https://github.com/seirasto
seirasto
MDQ6VXNlcjQyNTczMDg=
https://api.github.com/users/seirasto/orgs
https://api.github.com/users/seirasto/received_events
https://api.github.com/users/seirasto/repos
https://api.github.com/users/seirasto/starred{/owner}{/repo}
https://api.github.com/users/seirasto/subscriptions
User
https://api.github.com/users/seirasto
public
MEMBER
OK, thank you @seirasto for your hint.

That explains why you did not encounter the out of memory error: this only appears when the processing is distributed (on workers memory) and DirectRunner does not distribute the processing (all is done in a single machine). 
2022-06-30T09:28:22Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-1170984680
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps5Fy87o
https://api.github.com/repos/huggingface/datasets/issues/comments/1170984680/reactions
2022-06-30T09:33:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170984680
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
@albertvillanova Doesn't DirectRunner offer distributed processing through?

https://beam.apache.org/documentation/runners/direct/

```
Setting parallelism

Number of threads or subprocesses is defined by setting the direct_num_workers pipeline option. From 2.22.0, direct_num_workers = 0 is supported. When direct_num_workers is set to 0, it will set the number of threads/subprocess to the number of cores of the machine where the pipeline is running.

Setting running mode

In Beam 2.19.0 and newer, you can use the direct_running_mode pipeline option to set the running mode. direct_running_mode can be one of ['in_memory', 'multi_threading', 'multi_processing'].

in_memory: Runner and workers’ communication happens in memory (not through gRPC). This is a default mode.

multi_threading: Runner and workers communicate through gRPC and each worker runs in a thread.

multi_processing: Runner and workers communicate through gRPC and each worker runs in a subprocess.
```
2024-03-25T18:20:28Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-2018631688
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps54UeAI
https://api.github.com/repos/huggingface/datasets/issues/comments/2018631688/reactions
2024-03-25T18:20:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2018631688
https://avatars.githubusercontent.com/u/32650604?v=4
https://api.github.com/users/jdwillard19/events{/privacy}
https://api.github.com/users/jdwillard19/followers
https://api.github.com/users/jdwillard19/following{/other_user}
https://api.github.com/users/jdwillard19/gists{/gist_id}
https://github.com/jdwillard19
jdwillard19
MDQ6VXNlcjMyNjUwNjA0
https://api.github.com/users/jdwillard19/orgs
https://api.github.com/users/jdwillard19/received_events
https://api.github.com/users/jdwillard19/repos
https://api.github.com/users/jdwillard19/starred{/owner}{/repo}
https://api.github.com/users/jdwillard19/subscriptions
User
https://api.github.com/users/jdwillard19
public
MEMBER
Unrelated to the OOM issue, but we deprecated datasets with Beam scripts in #6474. I think we can close this issue
2024-03-26T12:54:44Z
https://github.com/huggingface/datasets/issues/4525#issuecomment-2020352858
https://api.github.com/repos/huggingface/datasets/issues/4525
IC_kwDODunzps54bCNa
https://api.github.com/repos/huggingface/datasets/issues/comments/2020352858/reactions
2024-03-26T12:54:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2020352858
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
While running the preprocessing of the natural_question dataset (see PR #4368), there is an issue for the "default" config (train+dev files).

Previously we ran the preprocessing for the "dev" config (only dev files) with success.

Train data files are larger than dev ones and apparently workers run out of memory while processing them.

Any help/hint is welcome!

Error message:
```
Data channel closed, unable to receive additional data from SDK sdk-0-0
```

Info from the Diagnostics tab:
```
Out of memory: Killed process 1882 (python) total-vm:6041764kB, anon-rss:3290928kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:9520kB oom_score_adj:900
The worker VM had to shut down one or more processes due to lack of memory.
```

## Additional information

### Stack trace
```
Traceback (most recent call last):
  File "/home/albert_huggingface_co/natural_questions/venv/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/home/albert_huggingface_co/natural_questions/venv/lib/python3.9/site-packages/datasets/commands/datasets_cli.py", line 39, in main
    service.run()
  File "/home/albert_huggingface_co/natural_questions/venv/lib/python3.9/site-packages/datasets/commands/run_beam.py", line 127, in run
    builder.download_and_prepare(
  File "/home/albert_huggingface_co/natural_questions/venv/lib/python3.9/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/albert_huggingface_co/natural_questions/venv/lib/python3.9/site-packages/datasets/builder.py", line 1389, in _download_and_prepare
    pipeline_results.wait_until_finish()
  File "/home/albert_huggingface_co/natural_questions/venv/lib/python3.9/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1667, in wait_until_finish
    raise DataflowRuntimeException(
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Data channel closed, unable to receive additional data from SDK sdk-0-0
```

### Logs
```
Error message from worker: Data channel closed, unable to receive additional data from SDK sdk-0-0

Workflow failed. Causes: S30:train/ReadAllFromText/ReadAllFiles/Reshard/ReshufflePerKey/GroupByKey/Read+train/ReadAllFromText/ReadAllFiles/Reshard/ReshufflePerKey/GroupByKey/GroupByWindow+train/ReadAllFromText/ReadAllFiles/Reshard/ReshufflePerKey/FlatMap(restore_timestamps)+train/ReadAllFromText/ReadAllFiles/Reshard/RemoveRandomKeys+train/ReadAllFromText/ReadAllFiles/ReadRange+train/Map(_parse_example)+train/Encode+train/Count N. Examples+train/Get values/Values+train/Save to parquet/Write/WriteImpl/WindowInto(WindowIntoFn)+train/Save to parquet/Write/WriteImpl/WriteBundles+train/Save to parquet/Write/WriteImpl/Pair+train/Save to parquet/Write/WriteImpl/GroupByKey/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: beamapp-alberthuggingface-06170554-5p23-harness-t4v9 Root cause: Data channel closed, unable to receive additional data from SDK sdk-0-0, beamapp-alberthuggingface-06170554-5p23-harness-t4v9 Root cause: The worker lost contact with the service., beamapp-alberthuggingface-06170554-5p23-harness-bwsj Root cause: The worker lost contact with the service., beamapp-alberthuggingface-06170554-5p23-harness-5052 Root cause: The worker lost contact with the service.
```
