Multiprocessed dataset builder
CONTRIBUTOR
I would also like to add a test, but am not sure whether it should go into `test_builder` (more natural imo) or `test_load` (which already contains a lot of the things I have to import to run my current testing setup). For reference, what I run to test that it works looks like:

```
import os
from pathlib import Path
import shutil

import datasets
from datasets.builder import DatasetBuilder
from datasets.features import Features, Value

DATASET_LOADING_SCRIPT_NAME = "__dummy_dataset1__"

DATASET_LOADING_SCRIPT_CODE = """
import os

import datasets
from datasets import DatasetInfo, Features, Split, SplitGenerator, Value


class __DummyDataset1__(datasets.GeneratorBasedBuilder):

    def _info(self) -> DatasetInfo:
        return DatasetInfo(features=Features({"text": Value("string")}))

    def _split_generators(self, dl_manager):
        return [
            SplitGenerator(Split.TRAIN, gen_kwargs={"filepaths": [os.path.join(dl_manager.manual_dir, "train1.txt"), os.path.join(dl_manager.manual_dir, "train2.txt")]}),
            SplitGenerator(Split.TEST, gen_kwargs={"filepaths": [os.path.join(dl_manager.manual_dir, "test.txt")]}),
        ]

    def _generate_examples(self, filepaths, **kwargs):
        idx = 0
        for filepath in filepaths:
            with open(filepath, "r", encoding="utf-8") as f:
                for line in f:
                    yield idx, {"text": line.strip()}
                    idx += 1
"""


def dataset_loading_script_dir(tmp_path):
    script_name = DATASET_LOADING_SCRIPT_NAME
    script_dir = tmp_path / script_name
    script_dir.mkdir()
    script_path = script_dir / f"{script_name}.py"
    with open(script_path, "w") as f:
        f.write(DATASET_LOADING_SCRIPT_CODE)
    return str(script_dir)


def data_dir(tmp_path):
    data_dir = tmp_path / "data_dir"
    data_dir.mkdir()
    with open(data_dir / "train1.txt", "w") as f:
        f.write("foo\n" * 10)
    with open(data_dir / "train2.txt", "w") as f:
        f.write("foo\n" * 10)
    with open(data_dir / "test.txt", "w") as f:
        f.write("bar\n" * 10)
    return str(data_dir)


def load_dataset_builder_multiprocessed(tmp_path):
    builder = datasets.load_dataset_builder(
        os.path.join(dataset_loading_script_dir(tmp_path), DATASET_LOADING_SCRIPT_NAME + ".py"),
        data_dir=data_dir(tmp_path),
    )
    assert isinstance(builder, DatasetBuilder)
    assert builder.name == DATASET_LOADING_SCRIPT_NAME
    assert builder.info.features == Features({"text": Value("string")})
    builder.download_and_prepare(tmp_path / "prepare_target", max_shard_size=500, num_proc=2)

if __name__ == "__main__":
    tmp_path = "tmp"
    if os.path.exists(tmp_path):
        raise FileExistsError(f"path {tmp_path} already exists")
    os.makedirs(tmp_path)
    try:
        load_dataset_builder_multiprocessed(Path(tmp_path))
    finally:
        # pass
        shutil.rmtree(tmp_path)
```
2022-10-12T20:01:14Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1276671611
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MGHZ7
https://api.github.com/repos/huggingface/datasets/issues/comments/1276671611/reactions
2022-10-12T20:01:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1276671611
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5107). All of your documentation changes will be reflected on that endpoint.
2022-10-12T20:08:26Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1276678012
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MGI98
https://api.github.com/repos/huggingface/datasets/issues/comments/1276678012/reactions
2022-10-12T20:08:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1276678012
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Nice ! I think the test can go in `test_builder.py` :)
2022-10-13T09:34:29Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1277317835
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MIlLL
https://api.github.com/repos/huggingface/datasets/issues/comments/1277317835/reactions
2022-10-13T09:34:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1277317835
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I've added sharded arrow dataset loading. Two WIP items in the PR:
- ~~Order is not conserved (it seems like the sharded files are read in the wrong order)~~
- the tqdm for preparing the splits is wrong (it compares against the size of the whole split rather than against the size of the multiprocessing shard, but I am not sure how to access the latter)

Also `naming.filenames_for_dataset_split` is not very elegant imo.

@lvwerra if you don't care about order, as I do, it's functional for now but I'd still quite like to get to the bottom of this.
2022-10-19T10:34:22Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1283789013
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MhRDV
https://api.github.com/repos/huggingface/datasets/issues/comments/1283789013/reactions
2022-10-19T11:53:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1283789013
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
CONTRIBUTOR
Found the ordering bug ! (`glob.glob` returning stuff in arbitrary order)
2022-10-19T11:52:43Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1283886213
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MhoyF
https://api.github.com/repos/huggingface/datasets/issues/comments/1283886213/reactions
2022-10-19T11:52:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1283886213
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
CONTRIBUTOR
I fixed the tqdm to be less misleading, but it can't tell where to stop. I am a bit hesitant to add a top-level tqdm (on the shard iterator) since for most intents it will do 0 -> N shards straight, but I am not sure what is the best way to present that info here.
2022-10-19T13:14:25Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1283995953
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MiDkx
https://api.github.com/repos/huggingface/datasets/issues/comments/1283995953/reactions
2022-10-19T13:14:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1283995953
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
MEMBER
I'm continuing the PR :)
2022-10-20T14:09:00Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1285615877
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5MoPEF
https://api.github.com/repos/huggingface/datasets/issues/comments/1285615877/reactions
2022-10-20T14:09:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1285615877
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Did a few changes:
- make shards naming consistent:
  - use `{builder_name}-{split_name}.{file_format}` when there's only 1 shard
  - otherwise use `{builder_name}-{split_name}-{shard_idx:05d}-of-{num_shards:05d}.{file_format}`
- update the reader to support reading several shards
  - added a new `shard_lengths` field in `SplitInfo` (FYI it is saved in `dataset_info.json` next to the shards as usual)
    - it's None when there's only 1 shard
    - otherwise it's a list of integers that correspond to the number of rows per shard
  - implemented partial reading to only memory map the required shards
    - e.g. when someone asks for a partial split like `train[:10%]`
- align the sharding for beam datasets
  - no more combining into 1 big arrow file
- added a tqdm bar
  - only one single bar, handled by the main process
  - gathers progress updates from other processes using `iflatmap_unordered`
  - shows the number of examples (even for datasets prepared by generating arrow tables)
- disabled multiprocessing by default - users must pass `num_proc` explicitly
- tests
- docs
2022-11-02T17:07:14Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1300930135
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5Nip5X
https://api.github.com/repos/huggingface/datasets/issues/comments/1300930135/reactions
2022-11-03T13:33:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1300930135
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright this is ready for review - sorry it ended up so big ^^'

If I can do anything to make it easier for your to review this PR @mariosasko let me know
2022-11-02T19:17:03Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1301102112
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5NjT4g
https://api.github.com/repos/huggingface/datasets/issues/comments/1301102112/reactions
2022-11-02T19:17:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1301102112
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Multiprocessing is disabled by default but we may show a warning to encourage users to pass `num_proc` if the dataset is split in many files. Let me know what you think
2022-11-03T13:45:16Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1302138338
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5NnQ3i
https://api.github.com/repos/huggingface/datasets/issues/comments/1302138338/reactions
2022-11-03T13:45:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1302138338
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hey, is this error seems to you guys natural? 

The package built from `0d4e3907` commit tag, and here is the version displayed from the import ... 
```bash
>>> datasets.__version__
'2.6.1.dev0'
>>> 
```

```bash
>>> data = load_dataset('dataset_loaders/rfw2latentplay', num_proc=14)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/somewhere//mambaforge/envs/datasets/lib/python3.8/site-packages/datasets/load.py", line 1719, in load_dataset
    builder_instance = load_dataset_builder(
  File "/somewhere//mambaforge/envs/datasets/lib/python3.8/site-packages/datasets/load.py", line 1523, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/somewhere//mambaforge/envs/datasets/lib/python3.8/site-packages/datasets/builder.py", line 1292, in __init__
    super().__init__(*args, **kwargs)
  File "/somewhere//mambaforge/envs/datasets/lib/python3.8/site-packages/datasets/builder.py", line 303, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/somewhere//mambaforge/envs/datasets/lib/python3.8/site-packages/datasets/builder.py", line 456, in _create_builder_config
    builder_config = self.BUILDER_CONFIG_CLASS(**config_kwargs)
TypeError: __init__() got an unexpected keyword argument 'num_proc'
```

Let me know if I can help fixing this ... 

2022-11-06T12:14:32Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1304787712
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5NxXsA
https://api.github.com/repos/huggingface/datasets/issues/comments/1304787712/reactions
2022-11-06T12:30:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1304787712
https://avatars.githubusercontent.com/u/11356471?v=4
https://api.github.com/users/parsa-ra/events{/privacy}
https://api.github.com/users/parsa-ra/followers
https://api.github.com/users/parsa-ra/following{/other_user}
https://api.github.com/users/parsa-ra/gists{/gist_id}
https://github.com/parsa-ra
parsa-ra
MDQ6VXNlcjExMzU2NDcx
https://api.github.com/users/parsa-ra/orgs
https://api.github.com/users/parsa-ra/received_events
https://api.github.com/users/parsa-ra/repos
https://api.github.com/users/parsa-ra/starred{/owner}{/repo}
https://api.github.com/users/parsa-ra/subscriptions
User
https://api.github.com/users/parsa-ra
public
MEMBER
> Do we have some benchmarks to see the speed-up?

On my machine running `load_dataset("oscar-corpus/OSCAR-2201", "br")` (which is split in shards) I go from 2-3k examples per sec to 4-5k examples per sec with num_proc=2 😉
2022-11-07T18:29:01Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1306019973
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5N2EiF
https://api.github.com/repos/huggingface/datasets/issues/comments/1306019973/reactions
2022-11-07T18:29:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1306019973
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Hey, is this error seems to you guys natural?
>
> The package built from 0d4e3907 commit tag, and here is the version displayed from the import ...

I don't know where you got the `0d4e3907` commit tag from, it doesn't seem to be in this PR. You should try installing from this PR, or wait for it to be merged on `main`
2022-11-07T18:29:56Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1306020872
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5N2EwI
https://api.github.com/repos/huggingface/datasets/issues/comments/1306020872/reactions
2022-11-07T18:29:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1306020872
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
## Splits vs Shards

Maybe it's a good idea to add some documentation on the `sharding` that can be achieved by passing `list` based arguments to the `SplitGenerator`s `gen_kwargs` ... 

I had to read the whole dataset generation source code to find this out ... 



2022-11-09T10:05:34Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1308509873
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5N_kax
https://api.github.com/repos/huggingface/datasets/issues/comments/1308509873/reactions
2022-11-09T10:06:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1308509873
https://avatars.githubusercontent.com/u/11356471?v=4
https://api.github.com/users/parsa-ra/events{/privacy}
https://api.github.com/users/parsa-ra/followers
https://api.github.com/users/parsa-ra/following{/other_user}
https://api.github.com/users/parsa-ra/gists{/gist_id}
https://github.com/parsa-ra
parsa-ra
MDQ6VXNlcjExMzU2NDcx
https://api.github.com/users/parsa-ra/orgs
https://api.github.com/users/parsa-ra/received_events
https://api.github.com/users/parsa-ra/repos
https://api.github.com/users/parsa-ra/starred{/owner}{/repo}
https://api.github.com/users/parsa-ra/subscriptions
User
https://api.github.com/users/parsa-ra
public
MEMBER
> Maybe it's a good idea to add some documentation on the sharding that can be achieved by passing list based arguments to the SplitGenerators gen_kwargs ...

This is part of this PR :) you can check the changes in docs/source/dataset_script.mdx
2022-11-09T11:01:47Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1308578680
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5N_1N4
https://api.github.com/repos/huggingface/datasets/issues/comments/1308578680/reactions
2022-11-09T11:01:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1308578680
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I took your comments into account @mariosasko thanks !
Let me know if it's good for you now ;)
2022-11-09T13:52:18Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1308798059
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5OAqxr
https://api.github.com/repos/huggingface/datasets/issues/comments/1308798059/reactions
2022-11-09T13:52:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1308798059
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
The doc CI should be fixed by now hopefully, merging !
2022-11-09T17:11:38Z
https://github.com/huggingface/datasets/pull/5107#issuecomment-1309076213
https://api.github.com/repos/huggingface/datasets/issues/5107
IC_kwDODunzps5OBur1
https://api.github.com/repos/huggingface/datasets/issues/comments/1309076213/reactions
2022-11-09T17:11:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1309076213
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This PR adds the multiprocessing part of #2650 (but not the caching of already-computed arrow files). On the other side, loading of sharded arrow files still needs to be implemented (sharded parquet files can already be loaded). 