Dataset.from_generator raises with sharded gen_args
COLLABORATOR
`gen_kwargs` should be a `dict`, as stated in the docstring, but you are passing a `list`.

So, to fix the error, replace the list of dicts with a dict of lists (and slightly modify the generator function):
```python
from pathlib import Path
import datasets

def process_yaml(files):
    for f in files:
        # process
        yield dict(...)


if __name__ == '__main__':
    import sys
    dir = Path(sys.argv[0]).parent
    ds = datasets.Dataset.from_generator(process_yaml, gen_kwargs={'files': [f for f in dir.glob('*.yml')]})
    ds.to_json('training.jsonl')
```
2023-10-02T15:17:11Z
https://github.com/huggingface/datasets/issues/6270#issuecomment-1743214085
https://api.github.com/repos/huggingface/datasets/issues/6270
IC_kwDODunzps5n51YF
https://api.github.com/repos/huggingface/datasets/issues/comments/1743214085/reactions
2023-10-02T15:17:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1743214085
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
That runs, and because my dataset is small, it's what I did to get past the problem.
However, it does not produce a sharded dataset.  From the doc string I expect there ought to be a way to call from_generator such that num_shards in the resulting data set is equal to the number of items in the list.
The part of the doc string that your suggestion is not responsive to is:
`            You can define a sharded dataset by passing the list of shards in *g
en_kwargs*.
`

What your suggestion does is calls the generator once, with the list argument, and produces a single shard dataset.

2023-10-02T15:35:17Z
https://github.com/huggingface/datasets/issues/6270#issuecomment-1743247280
https://api.github.com/repos/huggingface/datasets/issues/6270
IC_kwDODunzps5n59ew
https://api.github.com/repos/huggingface/datasets/issues/comments/1743247280/reactions
2023-10-02T15:35:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1743247280
https://avatars.githubusercontent.com/u/53510?v=4
https://api.github.com/users/hartmans/events{/privacy}
https://api.github.com/users/hartmans/followers
https://api.github.com/users/hartmans/following{/other_user}
https://api.github.com/users/hartmans/gists{/gist_id}
https://github.com/hartmans
hartmans
MDQ6VXNlcjUzNTEw
https://api.github.com/users/hartmans/orgs
https://api.github.com/users/hartmans/received_events
https://api.github.com/users/hartmans/repos
https://api.github.com/users/hartmans/starred{/owner}{/repo}
https://api.github.com/users/hartmans/subscriptions
User
https://api.github.com/users/hartmans
public
COLLABORATOR
The sharding mentioned here refers to using this function with `num_proc` (multiprocessing splits the `kwargs` into shards and passes them to the generator function)

> That runs, and because my dataset is small, it's what I did to get past the problem.

`from_generator` generates a memory-mapped dataset (can be larger than RAM), so the dataset size should not be an issue unless the generator function's implementation does not properly free the memory.

2023-10-02T15:53:43Z
https://github.com/huggingface/datasets/issues/6270#issuecomment-1743278420
https://api.github.com/repos/huggingface/datasets/issues/6270
IC_kwDODunzps5n6FFU
https://api.github.com/repos/huggingface/datasets/issues/comments/1743278420/reactions
2023-10-02T15:53:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1743278420
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
It sounds like you are saying that num_proc affects the form of gen_kwargs.
Are you saying that for non-zero num_proc gen_kwargs should be a list whose length is the same as num_proc?
Or are you saying that for non-zero num_proc, gen_kwargs should be a dict whose elements are lists the length of num_proc?

2023-10-02T16:00:39Z
https://github.com/huggingface/datasets/issues/6270#issuecomment-1743291143
https://api.github.com/repos/huggingface/datasets/issues/6270
IC_kwDODunzps5n6IMH
https://api.github.com/repos/huggingface/datasets/issues/comments/1743291143/reactions
2023-10-02T16:00:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1743291143
https://avatars.githubusercontent.com/u/53510?v=4
https://api.github.com/users/hartmans/events{/privacy}
https://api.github.com/users/hartmans/followers
https://api.github.com/users/hartmans/following{/other_user}
https://api.github.com/users/hartmans/gists{/gist_id}
https://github.com/hartmans
hartmans
MDQ6VXNlcjUzNTEw
https://api.github.com/users/hartmans/orgs
https://api.github.com/users/hartmans/received_events
https://api.github.com/users/hartmans/repos
https://api.github.com/users/hartmans/starred{/owner}{/repo}
https://api.github.com/users/hartmans/subscriptions
User
https://api.github.com/users/hartmans
public
CONTRIBUTOR
I ran some tests.  So, it looks like with num_proc greater than 1, gen_kwargs is expected to be a dict of lists.  It calls the generator also with a dict of lists, but the lists are split.
I.E. if my original has `gen_kwargs=dict(a=[0,1,2])`, then my generator might get called with `gen_kwalrgs=dict([0])`.
That all makes sense, but I definitely think there is room for improvement in the doc string here.
In order to suggest improvements to the doc string, I need to look at how the gen_kwargs are split, and figure out if:
* num_proc needs to exactly equal the length of the lists
* num_proc needs to evenly divide the length of the lists
* Or there's no required relationship.
I'll look into that and then propose an improved doc string if no one else gets to it first.
2023-10-02T16:18:04Z
https://github.com/huggingface/datasets/issues/6270#issuecomment-1743318926
https://api.github.com/repos/huggingface/datasets/issues/6270
IC_kwDODunzps5n6O-O
https://api.github.com/repos/huggingface/datasets/issues/comments/1743318926/reactions
2023-10-02T16:18:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1743318926
https://avatars.githubusercontent.com/u/53510?v=4
https://api.github.com/users/hartmans/events{/privacy}
https://api.github.com/users/hartmans/followers
https://api.github.com/users/hartmans/following{/other_user}
https://api.github.com/users/hartmans/gists{/gist_id}
https://github.com/hartmans
hartmans
MDQ6VXNlcjUzNTEw
https://api.github.com/users/hartmans/orgs
https://api.github.com/users/hartmans/received_events
https://api.github.com/users/hartmans/repos
https://api.github.com/users/hartmans/starred{/owner}{/repo}
https://api.github.com/users/hartmans/subscriptions
User
https://api.github.com/users/hartmans
public
CONTRIBUTOR
Okay, that was fun; I took a dive through the dataset code and feel like I have a much better understanding.
Here is my understanding of the behavior:
* max_proc is an upper limit on the number of shards that `from_generator` produces
* If `max_proc` is greater than 1, then all lists in *gen_kwargs* must be the same length
* If the lists in *gen_kwargs* are shorter than *num_proc* elements, *num_proc* will be reduced and a warning produced.  Put another way, `min(list_length, num_shards)` shards will be produced
* The members of the lists in *gen_kwargs* will be  partitioned among the created jobs.
To validate the above, take a look at
`_number_of_shards_in_gen_kwargs` and `_distribute_shards`  and `_split_gen_kwargs` in utils/sharding.py.
I've also chased down starting at *from_generator* all the way through to GeneratorBuilder and the calls to the functions in sharding.py.
Tomorrow I'll take a look at the contributing guidelines and see what's involved in putting together a PR to improve the doc string.
2023-10-03T01:21:39Z
https://github.com/huggingface/datasets/issues/6270#issuecomment-1744014920
https://api.github.com/repos/huggingface/datasets/issues/6270
IC_kwDODunzps5n845I
https://api.github.com/repos/huggingface/datasets/issues/comments/1744014920/reactions
2023-10-03T01:21:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1744014920
https://avatars.githubusercontent.com/u/53510?v=4
https://api.github.com/users/hartmans/events{/privacy}
https://api.github.com/users/hartmans/followers
https://api.github.com/users/hartmans/following{/other_user}
https://api.github.com/users/hartmans/gists{/gist_id}
https://github.com/hartmans
hartmans
MDQ6VXNlcjUzNTEw
https://api.github.com/users/hartmans/orgs
https://api.github.com/users/hartmans/received_events
https://api.github.com/users/hartmans/repos
https://api.github.com/users/hartmans/starred{/owner}{/repo}
https://api.github.com/users/hartmans/subscriptions
User
https://api.github.com/users/hartmans
public
### Describe the bug

According to the docs of Datasets.from_generator:
```
        gen_kwargs(`dict`, *optional*):
            Keyword arguments to be passed to the `generator` callable.
            You can define a sharded dataset by passing the list of shards in `gen_kwargs`.
```
So I'd expect that if gen_kwargs was a list, then my generator would be called once for each element in the list with the dict in the list for that element.
It doesn't work that way though.

### Steps to reproduce the bug

```python
#!/usr/bin/python

from pathlib import Path
import datasets

def process_yaml(file):
    yield dict(example=42)


if __name__ == '__main__':
    import sys
    dir = Path(sys.argv[0]).parent
    ds = datasets.Dataset.from_generator(process_yaml, gen_kwargs=[{'file':f} for f in dir.glob('*.yml')],
        )
    ds.to_json('training.jsonl')
    

```
```
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File "/tmp/dataset_bug.py", line 13, in <module>
    ds = datasets.Dataset.from_generator(process_yaml, gen_kwargs=[{'file':f} for f in dir.glob('*.yml')],
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1072, in from_generator
    ).read()
      ^^^^^^
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/io/generator.py", line 47, in read
    self.builder.download_and_prepare(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1555, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1656, in _prepare_split_single
    generator = self._generate_examples(**gen_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: datasets.packaged_modules.generator.generator.Generator._generate_examples() argument after ** must be a ```
mapping, not list


### Expected behavior

I would expect that process_yaml would be called once for each yaml file in the directory where the script is run.
I also tried with the list being in gen_kwargs, but in that case process_yaml gets called with a list.


### Environment info

- `datasets` version: 2.14.6.dev0 (git commit 0cc77d7f45c7369; also tested with 2.14.0)
- Platform: Linux-6.1.0-10-amd64-x86_64-with-glibc2.36
- Python version: 3.11.2
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
