Can't align optional features of struct
CONTRIBUTOR
Friendly bump, I would be happy to work on this issue once I get the go-ahead from the dev team. 
2023-12-11T14:38:33Z
https://github.com/huggingface/datasets/issues/6466#issuecomment-1850209835
https://api.github.com/repos/huggingface/datasets/issues/6466
IC_kwDODunzps5uR_Yr
https://api.github.com/repos/huggingface/datasets/issues/comments/1850209835/reactions
2023-12-11T14:38:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1850209835
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
NONE
Thanks for the PR!

I'm struggling with this as well and would love to see this PR merged. My case is slightly different, with keys completely missing rather than being `None`:

```
ds = Dataset.from_dict({'speaker': [{'name': 'Ben'}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

I would expect this to work as well because other Dataset functions already handle this situation well. For example, this works just as expected:

```
ds = Dataset.from_dict({'n': [1,2]})
ds_mapped = ds.map(lambda x: {
    'speaker': {'name': 'Ben'} if x['n'] == 1 else {'name': 'Fred', 'email': 'abc@aol.com'}
})
print(ds_mapped)
```
2024-01-24T05:14:40Z
https://github.com/huggingface/datasets/issues/6466#issuecomment-1907392557
https://api.github.com/repos/huggingface/datasets/issues/6466
IC_kwDODunzps5xsIAt
https://api.github.com/repos/huggingface/datasets/issues/comments/1907392557/reactions
2024-01-24T05:14:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1907392557
https://avatars.githubusercontent.com/u/133922112?v=4
https://api.github.com/users/vova-cyberhaven/events{/privacy}
https://api.github.com/users/vova-cyberhaven/followers
https://api.github.com/users/vova-cyberhaven/following{/other_user}
https://api.github.com/users/vova-cyberhaven/gists{/gist_id}
https://github.com/vova-cyberhaven
vova-cyberhaven
U_kgDOB_t9QA
https://api.github.com/users/vova-cyberhaven/orgs
https://api.github.com/users/vova-cyberhaven/received_events
https://api.github.com/users/vova-cyberhaven/repos
https://api.github.com/users/vova-cyberhaven/starred{/owner}{/repo}
https://api.github.com/users/vova-cyberhaven/subscriptions
User
https://api.github.com/users/vova-cyberhaven
public
CONTRIBUTOR
@vova-cyberhaven can you check with the new release if it fixes your issue? 
2024-02-15T15:19:32Z
https://github.com/huggingface/datasets/issues/6466#issuecomment-1946307879
https://api.github.com/repos/huggingface/datasets/issues/6466
IC_kwDODunzps50Ak0n
https://api.github.com/repos/huggingface/datasets/issues/comments/1946307879/reactions
2024-02-15T15:19:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1946307879
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`




### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.