nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
MEMBER
In 0.4.0, the assertion in `concatenate_datasets ` is on the features, and not the schema.
Could you try to update `nlp` ?

Also, since 0.4.0, you can use `dset_wikipedia.cast_(dset_books.features)` to avoid the schema cast hack.
2020-08-12T10:32:50Z
https://github.com/huggingface/datasets/issues/492#issuecomment-672792634
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY3Mjc5MjYzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/672792634/reactions
2020-08-12T10:32:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672792634
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Or maybe the assertion comes from elsewhere ?
2020-08-12T10:34:09Z
https://github.com/huggingface/datasets/issues/492#issuecomment-672793193
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY3Mjc5MzE5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/672793193/reactions
2020-08-12T10:34:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672793193
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I'm using the master branch. The assertion failure comes from the underlying `pa.concat_tables()`, which is in the pyarrow package. That method does check schemas.

Since `features.type` does not contain information about nullable vs non-nullable features, the `cast_()` method won't resolve the schema mismatch. There is information in a schema which is not stored in features.
2020-08-12T22:57:46Z
https://github.com/huggingface/datasets/issues/492#issuecomment-673149955
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY3MzE0OTk1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/673149955/reactions
2020-08-12T22:59:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/673149955
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
MEMBER
I'm doing a refactor of type inference in #363 . Both text fields should match after that
2020-08-20T15:04:49Z
https://github.com/huggingface/datasets/issues/492#issuecomment-677721325
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY3NzcyMTMyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/677721325/reactions
2020-08-20T15:04:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/677721325
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
By default nullable will be set to True
2020-08-20T15:05:14Z
https://github.com/huggingface/datasets/issues/492#issuecomment-677721541
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY3NzcyMTU0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/677721541/reactions
2020-08-20T15:05:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/677721541
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It should be good now. I was able to run

```python
>>> from nlp import concatenate_datasets, load_dataset
>>>
>>> bookcorpus = load_dataset("bookcorpus", split="train")
>>> wiki = load_dataset("wikipedia", "20200501.en", split="train")
>>> wiki.remove_columns_("title")  # only keep the text
>>>
>>> assert bookcorpus.features.type == wiki.features.type
>>> bert_dataset = concatenate_datasets([bookcorpus, wiki])
```
2020-08-26T12:36:09Z
https://github.com/huggingface/datasets/issues/492#issuecomment-680852269
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY4MDg1MjI2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/680852269/reactions
2020-08-26T12:36:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/680852269
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks!
2020-08-26T16:17:19Z
https://github.com/huggingface/datasets/issues/492#issuecomment-680980482
https://api.github.com/repos/huggingface/datasets/issues/492
MDEyOklzc3VlQ29tbWVudDY4MDk4MDQ4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/680980482/reactions
2020-08-26T16:17:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/680980482
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```
