pyarrow.lib.ArrowInvalid: Unable to merge: Field <field> has incompatible types
COLLABORATOR
Hi! The link pointing to the code that generated the dataset is broken. Can you please fix it to make debugging easier?
2023-04-04T14:22:08Z
https://github.com/huggingface/datasets/issues/5692#issuecomment-1496063375
https://api.github.com/repos/huggingface/datasets/issues/5692
IC_kwDODunzps5ZLB2P
https://api.github.com/repos/huggingface/datasets/issues/comments/1496063375/reactions
2023-04-04T14:22:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496063375
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> Hi! The link pointing to the code that generated the dataset is broken. Can you please fix it to make debugging easier?

Sorry about that, it's fixed now.

2023-04-04T14:35:30Z
https://github.com/huggingface/datasets/issues/5692#issuecomment-1496085464
https://api.github.com/repos/huggingface/datasets/issues/5692
IC_kwDODunzps5ZLHPY
https://api.github.com/repos/huggingface/datasets/issues/comments/1496085464/reactions
2023-04-04T14:38:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496085464
https://avatars.githubusercontent.com/u/32219669?v=4
https://api.github.com/users/cyanic-selkie/events{/privacy}
https://api.github.com/users/cyanic-selkie/followers
https://api.github.com/users/cyanic-selkie/following{/other_user}
https://api.github.com/users/cyanic-selkie/gists{/gist_id}
https://github.com/cyanic-selkie
cyanic-selkie
MDQ6VXNlcjMyMjE5NjY5
https://api.github.com/users/cyanic-selkie/orgs
https://api.github.com/users/cyanic-selkie/received_events
https://api.github.com/users/cyanic-selkie/repos
https://api.github.com/users/cyanic-selkie/starred{/owner}{/repo}
https://api.github.com/users/cyanic-selkie/subscriptions
User
https://api.github.com/users/cyanic-selkie
public
NONE
@cyanic-selkie could you explain how you fixed it? I met the same error in loading other datasets, is it due to the version of the library enviroment? 
2023-09-07T10:51:18Z
https://github.com/huggingface/datasets/issues/5692#issuecomment-1709936353
https://api.github.com/repos/huggingface/datasets/issues/5692
IC_kwDODunzps5l647h
https://api.github.com/repos/huggingface/datasets/issues/comments/1709936353/reactions
2023-09-07T10:51:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1709936353
https://avatars.githubusercontent.com/u/15760556?v=4
https://api.github.com/users/MingsYang/events{/privacy}
https://api.github.com/users/MingsYang/followers
https://api.github.com/users/MingsYang/following{/other_user}
https://api.github.com/users/MingsYang/gists{/gist_id}
https://github.com/MingsYang
MingsYang
MDQ6VXNlcjE1NzYwNTU2
https://api.github.com/users/MingsYang/orgs
https://api.github.com/users/MingsYang/received_events
https://api.github.com/users/MingsYang/repos
https://api.github.com/users/MingsYang/starred{/owner}{/repo}
https://api.github.com/users/MingsYang/subscriptions
User
https://api.github.com/users/MingsYang
public
NONE
@MingsYang I never fixed it. If you're referring to my comment above, I only meant I fixed the link to my code.

Anyway, I managed to work around the issue by using `streaming` when loading the dataset.
2023-09-07T10:59:15Z
https://github.com/huggingface/datasets/issues/5692#issuecomment-1709946292
https://api.github.com/repos/huggingface/datasets/issues/5692
IC_kwDODunzps5l67W0
https://api.github.com/repos/huggingface/datasets/issues/comments/1709946292/reactions
2023-09-07T10:59:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1709946292
https://avatars.githubusercontent.com/u/32219669?v=4
https://api.github.com/users/cyanic-selkie/events{/privacy}
https://api.github.com/users/cyanic-selkie/followers
https://api.github.com/users/cyanic-selkie/following{/other_user}
https://api.github.com/users/cyanic-selkie/gists{/gist_id}
https://github.com/cyanic-selkie
cyanic-selkie
MDQ6VXNlcjMyMjE5NjY5
https://api.github.com/users/cyanic-selkie/orgs
https://api.github.com/users/cyanic-selkie/received_events
https://api.github.com/users/cyanic-selkie/repos
https://api.github.com/users/cyanic-selkie/starred{/owner}{/repo}
https://api.github.com/users/cyanic-selkie/subscriptions
User
https://api.github.com/users/cyanic-selkie
public
NONE
@cyanic-selkie Emm, I get it. I just tried to use a new version python enviroment, and it show no errors anymore.
2023-09-07T11:42:48Z
https://github.com/huggingface/datasets/issues/5692#issuecomment-1710004088
https://api.github.com/repos/huggingface/datasets/issues/5692
IC_kwDODunzps5l7Jd4
https://api.github.com/repos/huggingface/datasets/issues/comments/1710004088/reactions
2023-09-07T11:42:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1710004088
https://avatars.githubusercontent.com/u/15760556?v=4
https://api.github.com/users/MingsYang/events{/privacy}
https://api.github.com/users/MingsYang/followers
https://api.github.com/users/MingsYang/following{/other_user}
https://api.github.com/users/MingsYang/gists{/gist_id}
https://github.com/MingsYang
MingsYang
MDQ6VXNlcjE1NzYwNTU2
https://api.github.com/users/MingsYang/orgs
https://api.github.com/users/MingsYang/received_events
https://api.github.com/users/MingsYang/repos
https://api.github.com/users/MingsYang/starred{/owner}{/repo}
https://api.github.com/users/MingsYang/subscriptions
User
https://api.github.com/users/MingsYang
public
NONE
Upgrade pyarrow to the latest version solves this problem in my case.
2024-01-14T07:24:20Z
https://github.com/huggingface/datasets/issues/5692#issuecomment-1890870677
https://api.github.com/repos/huggingface/datasets/issues/5692
IC_kwDODunzps5wtGWV
https://api.github.com/repos/huggingface/datasets/issues/comments/1890870677/reactions
2024-01-14T07:24:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1890870677
https://avatars.githubusercontent.com/u/11371254?v=4
https://api.github.com/users/ThyrixYang/events{/privacy}
https://api.github.com/users/ThyrixYang/followers
https://api.github.com/users/ThyrixYang/following{/other_user}
https://api.github.com/users/ThyrixYang/gists{/gist_id}
https://github.com/ThyrixYang
ThyrixYang
MDQ6VXNlcjExMzcxMjU0
https://api.github.com/users/ThyrixYang/orgs
https://api.github.com/users/ThyrixYang/received_events
https://api.github.com/users/ThyrixYang/repos
https://api.github.com/users/ThyrixYang/starred{/owner}{/repo}
https://api.github.com/users/ThyrixYang/subscriptions
User
https://api.github.com/users/ThyrixYang
public
### Describe the bug

When loading the dataset [wikianc-en](https://huggingface.co/datasets/cyanic-selkie/wikianc-en) which I created using [this](https://github.com/cyanic-selkie/wikianc) code, I get the following error:

```
Traceback (most recent call last):
  File "/home/sven/code/rector/answer-detection/train.py", line 106, in <module>
    (dataset, weights) = get_dataset(args.dataset, tokenizer, labels, args.padding)
  File "/home/sven/code/rector/answer-detection/dataset.py", line 106, in get_dataset
    dataset = load_dataset("cyanic-selkie/wikianc-en")
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/load.py", line 1794, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/builder.py", line 1106, in as_dataset
    datasets = map_nested(
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 443, in map_nested
    mapped = [
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 444, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 346, in _single_map_nested
    return function(data_struct)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/builder.py", line 1136, in _build_single_dataset
    ds = self._as_dataset(
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/builder.py", line 1207, in _as_dataset
    dataset_kwargs = ArrowReader(cache_dir, self.info).read(
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 239, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 260, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 203, in _read_files
    pa_table = concat_tables(pa_tables) if len(pa_tables) != 1 else pa_tables[0]
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1808, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1514, in from_tables
    return cls.from_blocks(blocks)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1427, in from_blocks
    table = cls._concat_blocks(blocks, axis=0)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1373, in _concat_blocks
    return pa.concat_tables(pa_tables, promote=True)
  File "pyarrow/table.pxi", line 5224, in pyarrow.lib.concat_tables
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Unable to merge: Field paragraph_anchors has incompatible types: list<: struct<start: uint32 not null, end: uint32 not null, qid: uint32, pageid: uint32, title: string not null> not null> vs list<item: struct<start: uint32, end: uint32, qid: uint32, pageid: uint32, title: string>>
```

This only happens when I load the `train` split, indicating that the size of the dataset is the deciding factor. 

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("cyanic-selkie/wikianc-en", split="train")
```

### Expected behavior

The dataset should load normally without any errors.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-6.2.8-arch1-1-x86_64-with-glibc2.37
- Python version: 3.10.10
- PyArrow version: 11.0.0
- Pandas version: 1.5.3