Inconsistent "The features can't be aligned" error when combining map, multiprocessing, and variable length outputs
COLLABORATOR
This scenario currently requires explicitly passing the target features (to avoid the error): 
```python
import datasets

...

features = dataset.features
features["output"] = = [{"test": datasets.Value("int64")}]
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32, features=features)
```
2023-07-12T15:58:05Z
https://github.com/huggingface/datasets/issues/6020#issuecomment-1632803184
https://api.github.com/repos/huggingface/datasets/issues/6020
IC_kwDODunzps5hUplw
https://api.github.com/repos/huggingface/datasets/issues/comments/1632803184/reactions
2023-07-12T15:58:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1632803184
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I just encountered the same error in the same situation (multiprocessing with variable length outputs).

The funny (or dangerous?) thing is, that this error only showed up when testing with a small test dataset (16 examples, ValueError with `num_proc` >1) but the same code works fine for the full dataset (~70k examples).

@mariosasko Any idea on how to do that with a nested feature with lists of variable lengths containing dicts?

EDIT: Was able to narrow it down: >200 Examples: no error, <150 Examples: Error. 
Now idea what to make of this but pretty obvious that this is a bug....
2023-10-25T13:54:42Z
https://github.com/huggingface/datasets/issues/6020#issuecomment-1779330972
https://api.github.com/repos/huggingface/datasets/issues/6020
IC_kwDODunzps5qDm-c
https://api.github.com/repos/huggingface/datasets/issues/comments/1779330972/reactions
2023-10-25T14:14:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1779330972
https://avatars.githubusercontent.com/u/2862336?v=4
https://api.github.com/users/jphme/events{/privacy}
https://api.github.com/users/jphme/followers
https://api.github.com/users/jphme/following{/other_user}
https://api.github.com/users/jphme/gists{/gist_id}
https://github.com/jphme
jphme
MDQ6VXNlcjI4NjIzMzY=
https://api.github.com/users/jphme/orgs
https://api.github.com/users/jphme/received_events
https://api.github.com/users/jphme/repos
https://api.github.com/users/jphme/starred{/owner}{/repo}
https://api.github.com/users/jphme/subscriptions
User
https://api.github.com/users/jphme
public
NONE
This error also occurs while concatenating the datasets.
2024-02-10T19:24:28Z
https://github.com/huggingface/datasets/issues/6020#issuecomment-1937101333
https://api.github.com/repos/huggingface/datasets/issues/6020
IC_kwDODunzps5zddIV
https://api.github.com/repos/huggingface/datasets/issues/comments/1937101333/reactions
2024-02-10T19:24:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1937101333
https://avatars.githubusercontent.com/u/87112292?v=4
https://api.github.com/users/Ananthzeke/events{/privacy}
https://api.github.com/users/Ananthzeke/followers
https://api.github.com/users/Ananthzeke/following{/other_user}
https://api.github.com/users/Ananthzeke/gists{/gist_id}
https://github.com/Ananthzeke
Ananthzeke
MDQ6VXNlcjg3MTEyMjky
https://api.github.com/users/Ananthzeke/orgs
https://api.github.com/users/Ananthzeke/received_events
https://api.github.com/users/Ananthzeke/repos
https://api.github.com/users/Ananthzeke/starred{/owner}{/repo}
https://api.github.com/users/Ananthzeke/subscriptions
User
https://api.github.com/users/Ananthzeke
public
NONE
I'm running into the same error, is there any working workaround for this that doesnt involve using a larger subset or reducing the number of workers? I couldn't get the `features` set mentioned above to work...
2024-10-27T06:19:31Z
https://github.com/huggingface/datasets/issues/6020#issuecomment-2439871651
https://api.github.com/repos/huggingface/datasets/issues/6020
IC_kwDODunzps6RbXyj
https://api.github.com/repos/huggingface/datasets/issues/comments/2439871651/reactions
2024-10-27T06:30:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2439871651
https://avatars.githubusercontent.com/u/20804972?v=4
https://api.github.com/users/SirRob1997/events{/privacy}
https://api.github.com/users/SirRob1997/followers
https://api.github.com/users/SirRob1997/following{/other_user}
https://api.github.com/users/SirRob1997/gists{/gist_id}
https://github.com/SirRob1997
SirRob1997
MDQ6VXNlcjIwODA0OTcy
https://api.github.com/users/SirRob1997/orgs
https://api.github.com/users/SirRob1997/received_events
https://api.github.com/users/SirRob1997/repos
https://api.github.com/users/SirRob1997/starred{/owner}{/repo}
https://api.github.com/users/SirRob1997/subscriptions
User
https://api.github.com/users/SirRob1997
public
### Describe the bug

I'm using a dataset with map and multiprocessing to run a function that returned a variable length list of outputs. This output list may be empty. Normally this is handled fine, but there is an edge case that crops up when using multiprocessing. In some cases, an empty list result ends up in a dataset shard consisting of a single item. This results in a `The features can't be aligned` error that is difficult to debug because it depends on the number of processes/shards used.

I've reproduced a minimal example below. My current workaround is to fill empty results with a dummy value that I filter after, but this was a weird error that took a while to track down.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_list([{'idx':i} for i in range(60)])

def test_func(row, idx):
    if idx==58:
        return {'output': []}
    else:
        return {'output' : [{'test':1}, {'test':2}]}

# this works fine
test1 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=4)

# this fails
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32)
>ValueError: The features can't be aligned because the key output of features {'idx': Value(dtype='int64', id=None), 'output': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None)} has unexpected type - Sequence(feature=Value(dtype='null', id=None), length=-1, id=None) (expected either [{'test': Value(dtype='int64', id=None)}] or Value("null").
```

The error occurs during the check

```python
_check_if_features_can_be_aligned([dset.features for dset in dsets])
```

When the multiprocessing splitting lines up just right with the empty return value, one of the `dset` in `dsets` will have a single item with an empty list value, causing the error.

### Expected behavior

Expected behavior is the result would be the same regardless of the `num_proc` value used.

### Environment info

Datasets version 2.11.0
Python 3.9.16