Unwanted column padding in nested lists of dicts
MEMBER
Answer from @lhoestq:

> No
> This is because Arrow and Parquet a columnar format: they require a fixed type for each column. So if you have nested dicts, each item should have the same subfields

The way around I found is the handle it after sampling with this function:

```python
def remove_padding(example):
    if isinstance(example, list):
        return [remove_padding(value) if isinstance(value, (dict, list)) else value for value in example]
    elif isinstance(example, Mapping):
        return {
            key: remove_padding(value) if isinstance(value, (dict, list)) else value
            for key, value in example.items()
            if value is not None
        }
    else:
        raise TypeError("Input must be a list or a dictionary.")

# Example:
example = next(iter(dataset))
example = remove_padding(example)
```
2025-06-16T13:43:30Z
https://github.com/huggingface/datasets/issues/7617#issuecomment-2976720666
https://api.github.com/repos/huggingface/datasets/issues/7617
IC_kwDODunzps6xbSca
https://api.github.com/repos/huggingface/datasets/issues/comments/2976720666/reactions
2025-06-16T13:43:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2976720666
https://avatars.githubusercontent.com/u/45557362?v=4
https://api.github.com/users/qgallouedec/events{/privacy}
https://api.github.com/users/qgallouedec/followers
https://api.github.com/users/qgallouedec/following{/other_user}
https://api.github.com/users/qgallouedec/gists{/gist_id}
https://github.com/qgallouedec
qgallouedec
MDQ6VXNlcjQ1NTU3MzYy
https://api.github.com/users/qgallouedec/orgs
https://api.github.com/users/qgallouedec/received_events
https://api.github.com/users/qgallouedec/repos
https://api.github.com/users/qgallouedec/starred{/owner}{/repo}
https://api.github.com/users/qgallouedec/subscriptions
User
https://api.github.com/users/qgallouedec
public
```python
from datasets import Dataset

dataset = Dataset.from_dict({
    "messages": [
        [
            {"a": "...",},
            {"b": "...",},
        ],
    ]
})
print(dataset[0])
```

What I get:
```
{'messages': [{'a': '...', 'b': None}, {'a': None, 'b': '...'}]}
```

What I want:

```
{'messages': [{'a': '...'}, {'b': '...'}]}
```

Is there an easy way to automatically remove these auto-filled null/none values?

If not, I probably need a recursive none exclusion function, don't I?

Datasets 3.6.0