Sharding error with Multilingual LibriSpeech
MEMBER
Thanks for reporting, @Nithin-Holla.

This is a known issue for multiple datasets and we are investigating it:
- See e.g.: https://huggingface.co/datasets/ami/discussions/3
2023-01-10T06:52:57Z
https://github.com/huggingface/datasets/issues/5414#issuecomment-1376808468
https://api.github.com/repos/huggingface/datasets/issues/5414
IC_kwDODunzps5SEG4U
https://api.github.com/repos/huggingface/datasets/issues/comments/1376808468/reactions
2023-01-10T06:57:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1376808468
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Main issue:
- #5415
2023-01-10T07:42:33Z
https://github.com/huggingface/datasets/issues/5414#issuecomment-1376849352
https://api.github.com/repos/huggingface/datasets/issues/5414
IC_kwDODunzps5SEQ3I
https://api.github.com/repos/huggingface/datasets/issues/comments/1376849352/reactions
2023-01-10T07:42:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1376849352
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
@albertvillanova Thanks! As a workaround for now, can I use the dataset in streaming mode?
2023-01-10T07:54:10Z
https://github.com/huggingface/datasets/issues/5414#issuecomment-1376859528
https://api.github.com/repos/huggingface/datasets/issues/5414
IC_kwDODunzps5SETWI
https://api.github.com/repos/huggingface/datasets/issues/comments/1376859528/reactions
2023-01-10T07:54:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1376859528
https://avatars.githubusercontent.com/u/19574344?v=4
https://api.github.com/users/Nithin-Holla/events{/privacy}
https://api.github.com/users/Nithin-Holla/followers
https://api.github.com/users/Nithin-Holla/following{/other_user}
https://api.github.com/users/Nithin-Holla/gists{/gist_id}
https://github.com/Nithin-Holla
Nithin-Holla
MDQ6VXNlcjE5NTc0MzQ0
https://api.github.com/users/Nithin-Holla/orgs
https://api.github.com/users/Nithin-Holla/received_events
https://api.github.com/users/Nithin-Holla/repos
https://api.github.com/users/Nithin-Holla/starred{/owner}{/repo}
https://api.github.com/users/Nithin-Holla/subscriptions
User
https://api.github.com/users/Nithin-Holla
public
MEMBER
Yes, @Nithin-Holla, in the meantime you can use this dataset in streaming mode.
2023-01-10T09:46:37Z
https://github.com/huggingface/datasets/issues/5414#issuecomment-1376987671
https://api.github.com/repos/huggingface/datasets/issues/5414
IC_kwDODunzps5SEyoX
https://api.github.com/repos/huggingface/datasets/issues/comments/1376987671/reactions
2023-01-10T09:46:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1376987671
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Describe the bug

Loading the German Multilingual LibriSpeech dataset results in a RuntimeError regarding sharding with the following stacktrace:

```
Downloading and preparing dataset multilingual_librispeech/german to /home/nithin/datadrive/cache/huggingface/datasets/facebook___multilingual_librispeech/german/2.1.0/1904af50f57a5c370c9364cc337699cfe496d4e9edcae6648a96be23086362d0...
Downloading data files: 100%
3/3 [00:00<00:00, 107.23it/s]
Downloading data files: 100%
1/1 [00:00<00:00, 35.08it/s]
Downloading data files: 100%
6/6 [00:00<00:00, 303.36it/s]
Downloading data files: 100%
3/3 [00:00<00:00, 130.37it/s]
Downloading data files: 100%
1049/1049 [00:00<00:00, 4491.40it/s]
Downloading data files: 100%
37/37 [00:00<00:00, 1096.78it/s]
Downloading data files: 100%
40/40 [00:00<00:00, 1003.93it/s]
Extracting data files: 100%
3/3 [00:11<00:00, 2.62s/it]
Generating train split:
469942/0 [34:13<00:00, 273.21 examples/s]
Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-74fa6d092bdc> in <module>
----> 1 mls = load_dataset(MLS_DATASET, 
      2                    LANGUAGE,
      3                    cache_dir="~/datadrive/cache/huggingface/datasets",
      4                    ignore_verifications=True)

/anaconda/envs/py38_default/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   1755 
   1756     # Download and prepare data
-> 1757     builder_instance.download_and_prepare(
   1758         download_config=download_config,
   1759         download_mode=download_mode,

/anaconda/envs/py38_default/lib/python3.8/site-packages/datasets/builder.py in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    858                         if num_proc is not None:
    859                             prepare_split_kwargs["num_proc"] = num_proc
--> 860                         self._download_and_prepare(
    861                             dl_manager=dl_manager,
    862                             verify_infos=verify_infos,

/anaconda/envs/py38_default/lib/python3.8/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs)
   1609 
   1610     def _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs):
...
RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
    - key audio_archives has length 1049
    - key local_extracted_archive has length 1049
    - key limited_ids_paths has length 1
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.
```

### Steps to reproduce the bug

Here is the code to reproduce it:

```python
from datasets import load_dataset

MLS_DATASET = "facebook/multilingual_librispeech"
LANGUAGE = "german"

mls = load_dataset(MLS_DATASET, 
                   LANGUAGE, 
                   cache_dir="~/datadrive/cache/huggingface/datasets", 
                   ignore_verifications=True)
```

### Expected behavior

The expected behaviour is that the dataset is successfully loaded.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-1094-azure-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 10.0.1
- Pandas version: 1.2.4