to_pandas conversion doesn't always work
MEMBER
**Edit**: other topic previously in this message moved to a new issue: https://github.com/huggingface/nlp/issues/387
2020-07-10T23:04:08Z
https://github.com/huggingface/datasets/issues/376#issuecomment-656931379
https://api.github.com/repos/huggingface/datasets/issues/376
MDEyOklzc3VlQ29tbWVudDY1NjkzMTM3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/656931379/reactions
2020-07-14T06:24:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656931379
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Could you try to update pyarrow to >=0.17.0 ? It should fix the `to_pandas` bug

Also I'm not sure that structures like list<struct> are fully supported in the lib (none of the datasets use that).
It can cause issues when using dataset transforms like `filter` for example
2020-07-13T09:51:25Z
https://github.com/huggingface/datasets/issues/376#issuecomment-657454597
https://api.github.com/repos/huggingface/datasets/issues/376
MDEyOklzc3VlQ29tbWVudDY1NzQ1NDU5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/657454597/reactions
2020-07-13T09:51:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/657454597
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
For some complex nested types, the conversion from Arrow to python dict through pandas doesn't seem to be possible.

Here is an example using the official SQUAD v2 JSON file.

This example was found while investigating #373.

```python
>>> squad = load_dataset('json', data_files={nlp.Split.TRAIN: ["./train-v2.0.json"]}, download_mode=nlp.GenerateMode.FORCE_REDOWNLOAD, version="1.0.0", field='data')
>>> squad['train']
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 442)
>>> squad['train'][0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/thomwolf/Documents/GitHub/datasets/src/nlp/arrow_dataset.py", line 589, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/Users/thomwolf/Documents/GitHub/datasets/src/nlp/arrow_dataset.py", line 529, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
```

cc @lhoestq would we have a way to detect this from the schema maybe?

Here is the schema for this pretty complex JSON:
```python
>>> squad['train'].schema
title: string
paragraphs: list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>
  child 0, item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
      child 0, qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>
          child 0, item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>
              child 0, question: string
              child 1, id: string
              child 2, answers: list<item: struct<text: string, answer_start: int64>>
                  child 0, item: struct<text: string, answer_start: int64>
                      child 0, text: string
                      child 1, answer_start: int64
              child 3, is_impossible: bool
              child 4, plausible_answers: list<item: struct<text: string, answer_start: int64>>
                  child 0, item: struct<text: string, answer_start: int64>
                      child 0, text: string
                      child 1, answer_start: int64
      child 1, context: string
```