Don't do unnecessary list type casting to avoid replacing None values by empty lists
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-05-04T16:47:56Z
https://github.com/huggingface/datasets/pull/4282#issuecomment-1117577038
https://api.github.com/repos/huggingface/datasets/issues/4282
IC_kwDODunzps5CnN9O
https://api.github.com/repos/huggingface/datasets/issues/comments/1117577038/reactions
2022-05-06T10:43:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117577038
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
Quick question about the message in the warning. You say "will be fixed in a future major version" but don't you mean "will raise an error in a future major version"?
2022-05-04T18:59:43Z
https://github.com/huggingface/datasets/pull/4282#issuecomment-1117696430
https://api.github.com/repos/huggingface/datasets/issues/4282
IC_kwDODunzps5CnrGu
https://api.github.com/repos/huggingface/datasets/issues/comments/1117696430/reactions
2022-05-04T18:59:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1117696430
https://avatars.githubusercontent.com/u/35901082?v=4
https://api.github.com/users/sgugger/events{/privacy}
https://api.github.com/users/sgugger/followers
https://api.github.com/users/sgugger/following{/other_user}
https://api.github.com/users/sgugger/gists{/gist_id}
https://github.com/sgugger
sgugger
MDQ6VXNlcjM1OTAxMDgy
https://api.github.com/users/sgugger/orgs
https://api.github.com/users/sgugger/received_events
https://api.github.com/users/sgugger/repos
https://api.github.com/users/sgugger/starred{/owner}{/repo}
https://api.github.com/users/sgugger/subscriptions
User
https://api.github.com/users/sgugger
public
MEMBER
Right ! Good catch, thanks, I updated the message to say "will raise an error in a future major version"
2022-05-05T12:21:47Z
https://github.com/huggingface/datasets/pull/4282#issuecomment-1118483441
https://api.github.com/repos/huggingface/datasets/issues/4282
IC_kwDODunzps5CqrPx
https://api.github.com/repos/huggingface/datasets/issues/comments/1118483441/reactions
2022-05-05T12:21:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118483441
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
In certain cases, `None` values are replaced by empty lists when casting feature types.

It happens every time you cast an array of nested lists like [None, [0, 1, 2, 3]] to a different type (to change the integer precision for example). In this case you'd get [[], [0, 1, 2, 3]] for example. This issue comes from PyArrow, see the discussion in https://github.com/huggingface/datasets/issues/3676

This issue also happens when no type casting is needed, because casting is supposed to be a no-op in this case. But as https://github.com/huggingface/datasets/issues/3676 shown, it's not the case and `None` are replaced by empty lists even if we cast to the exact same type.

In this PR I just workaround this bug in the case where no type casting is needed. In particular, I only call `pa.ListArray.from_arrays` only when necessary.

I also added a warning when some `None` are effectively replaced by empty lists. I wanted to raise an error in this case, but maybe we should wait a major update to do so

This PR fixes this particular case, that is occurring in `run_qa.py` in `transformers`:
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
# before:
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
#
# now:
#              b
# 0  [None, [0]]
# 1  [None, [0]]
# 2  [None, [0]]
# 3  [None, [0]]
```

cc @sgugger 