[Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type
CONTRIBUTOR
Here the full error trace:

```
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-1-7aaf3f011358> in <module>
      1 import nlp
      2 ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
----> 3 ds.map(lambda x: x, load_from_cache_file=False)

~/python_bin/nlp/arrow_dataset.py in map(self, function, with_indices, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, arrow_schema, disable_nullable)
    549
    550         if update_data:
--> 551             writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file
    552
    553             # Create new Dataset from buffer or file

~/python_bin/nlp/arrow_writer.py in finalize(self, close_stream)
    182     def finalize(self, close_stream=True):
    183         if self.pa_writer is not None:
--> 184             self.write_on_file()
    185             self.pa_writer.close()
    186         if close_stream:

~/python_bin/nlp/arrow_writer.py in write_on_file(self)
    104         """
    105         if self.current_rows:
--> 106             pa_array = pa.array(self.current_rows, type=self._type)
    107             first_example = pa.array(self.current_rows[0:1], type=self._type)[0]
    108             # Sanity check

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Could not convert TagMe with type str: converting to null type
```
2020-05-28T14:45:13Z
https://github.com/huggingface/datasets/issues/211#issuecomment-635395928
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDYzNTM5NTkyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635395928/reactions
2020-05-28T14:45:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635395928
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
Actually thinking a bit more about it, it's probably a data sample that is not correct in `trivia_qa`. But I'm a bit surprised though that we managed to write it in .arrow format and now cannot write it anymore after an "identity" mapping.
2020-05-28T14:59:16Z
https://github.com/huggingface/datasets/issues/211#issuecomment-635404495
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDYzNTQwNDQ5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/635404495/reactions
2020-05-28T14:59:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635404495
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I don't have this error :x
2020-05-28T16:10:19Z
https://github.com/huggingface/datasets/issues/211#issuecomment-635444482
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDYzNTQ0NDQ4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/635444482/reactions
2020-05-28T16:10:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635444482
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Interesting, maybe I have a very old cache of trivia_qa...thanks for checking
2020-05-28T16:18:33Z
https://github.com/huggingface/datasets/issues/211#issuecomment-635449838
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDYzNTQ0OTgzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635449838/reactions
2020-05-28T16:18:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635449838
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I'm running it right now on colab to double check
2020-05-28T16:19:30Z
https://github.com/huggingface/datasets/issues/211#issuecomment-635450376
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDYzNTQ1MDM3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/635450376/reactions
2020-05-28T16:19:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635450376
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Actually, I know what the problem is...I'm quite sure it's a bug. Here we take some test inputs: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L472

It might be that in the test inputs, a `Sequence` type value is an emtpy list. So in my case I have `ds[0]["entity_pages']["wiki_context"] = []`. => this leads to an `arrow_schema` equal to `null`  for `["entity_pages']["wiki_context"]` => see line: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L501  instead of list of string which it should for other examples. 

Guess it's an edge case, but it can happen.
2020-05-28T17:38:04Z
https://github.com/huggingface/datasets/issues/211#issuecomment-635493698
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDYzNTQ5MzY5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635493698/reactions
2020-05-28T17:38:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635493698
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Good point, I think the schema should be infered at the writing stage where we have a `writer_batch_size` number of examples (typically 10k) so it's even less likely to run into this scenario.
2020-06-20T21:10:54Z
https://github.com/huggingface/datasets/issues/211#issuecomment-647045809
https://api.github.com/repos/huggingface/datasets/issues/211
MDEyOklzc3VlQ29tbWVudDY0NzA0NTgwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/647045809/reactions
2020-06-20T21:10:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647045809
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
Running the following code 

```
import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, load_from_cache_file=False)
```

triggers a `ArrowInvalid: Could not convert TagMe with type str: converting to null type` error.

On the other hand if we remove a certain column of `trivia_qa` which seems responsible for the bug, it works:

```
import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, remove_columns=["entity_pages"], load_from_cache_file=False)
```

. Seems quite hard to debug what's going on here... @lhoestq @thomwolf - do you have a good first guess what the problem could be?

**Note** BTW: I think this could be a good test to check that the datasets work correctly: Take a tiny portion of the dataset and check that it can be written correctly.