load_dataset('the_pile_openwebtext2') produces ArrowInvalid, value too large to fit in C integer type
NONE
I encountered the same bug using different datasets.
any suggestions?
2021-11-01T17:18:26Z
https://github.com/huggingface/datasets/issues/3053#issuecomment-956425060
https://api.github.com/repos/huggingface/datasets/issues/3053
IC_kwDODunzps45AeNk
https://api.github.com/repos/huggingface/datasets/issues/comments/956425060/reactions
2021-11-01T17:18:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/956425060
https://avatars.githubusercontent.com/u/35503583?v=4
https://api.github.com/users/srulikbd/events{/privacy}
https://api.github.com/users/srulikbd/followers
https://api.github.com/users/srulikbd/following{/other_user}
https://api.github.com/users/srulikbd/gists{/gist_id}
https://github.com/srulikbd
srulikbd
MDQ6VXNlcjM1NTAzNTgz
https://api.github.com/users/srulikbd/orgs
https://api.github.com/users/srulikbd/received_events
https://api.github.com/users/srulikbd/repos
https://api.github.com/users/srulikbd/starred{/owner}{/repo}
https://api.github.com/users/srulikbd/subscriptions
User
https://api.github.com/users/srulikbd
public
NONE
+1, can reproduce here!
2021-11-08T21:48:38Z
https://github.com/huggingface/datasets/issues/3053#issuecomment-963605031
https://api.github.com/repos/huggingface/datasets/issues/3053
IC_kwDODunzps45b3In
https://api.github.com/repos/huggingface/datasets/issues/comments/963605031/reactions
2021-11-08T21:48:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/963605031
https://avatars.githubusercontent.com/u/813367?v=4
https://api.github.com/users/moinnadeem/events{/privacy}
https://api.github.com/users/moinnadeem/followers
https://api.github.com/users/moinnadeem/following{/other_user}
https://api.github.com/users/moinnadeem/gists{/gist_id}
https://github.com/moinnadeem
moinnadeem
MDQ6VXNlcjgxMzM2Nw==
https://api.github.com/users/moinnadeem/orgs
https://api.github.com/users/moinnadeem/received_events
https://api.github.com/users/moinnadeem/repos
https://api.github.com/users/moinnadeem/starred{/owner}{/repo}
https://api.github.com/users/moinnadeem/subscriptions
User
https://api.github.com/users/moinnadeem
public
CONTRIBUTOR
I get the same error
Platform: Windows 10
Python: python 3.8.8
PyArrow: 5.0
2021-12-21T10:12:37Z
https://github.com/huggingface/datasets/issues/3053#issuecomment-998647688
https://api.github.com/repos/huggingface/datasets/issues/3053
IC_kwDODunzps47hieI
https://api.github.com/repos/huggingface/datasets/issues/comments/998647688/reactions
2021-12-21T10:17:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/998647688
https://avatars.githubusercontent.com/u/68908804?v=4
https://api.github.com/users/apergo-ai/events{/privacy}
https://api.github.com/users/apergo-ai/followers
https://api.github.com/users/apergo-ai/following{/other_user}
https://api.github.com/users/apergo-ai/gists{/gist_id}
https://github.com/apergo-ai
apergo-ai
MDQ6VXNlcjY4OTA4ODA0
https://api.github.com/users/apergo-ai/orgs
https://api.github.com/users/apergo-ai/received_events
https://api.github.com/users/apergo-ai/repos
https://api.github.com/users/apergo-ai/starred{/owner}{/repo}
https://api.github.com/users/apergo-ai/subscriptions
User
https://api.github.com/users/apergo-ai
public
NONE
I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).
2022-10-17T19:48:01Z
https://github.com/huggingface/datasets/issues/3053#issuecomment-1281392422
https://api.github.com/repos/huggingface/datasets/issues/3053
IC_kwDODunzps5MYH8m
https://api.github.com/repos/huggingface/datasets/issues/comments/1281392422/reactions
2022-10-17T19:48:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1281392422
https://avatars.githubusercontent.com/u/50339167?v=4
https://api.github.com/users/tc-wolf/events{/privacy}
https://api.github.com/users/tc-wolf/followers
https://api.github.com/users/tc-wolf/following{/other_user}
https://api.github.com/users/tc-wolf/gists{/gist_id}
https://github.com/tc-wolf
tc-wolf
MDQ6VXNlcjUwMzM5MTY3
https://api.github.com/users/tc-wolf/orgs
https://api.github.com/users/tc-wolf/received_events
https://api.github.com/users/tc-wolf/repos
https://api.github.com/users/tc-wolf/starred{/owner}{/repo}
https://api.github.com/users/tc-wolf/subscriptions
User
https://api.github.com/users/tc-wolf
public
COLLABORATOR
Addressed in https://huggingface.co/datasets/the_pile_openwebtext2/discussions/4
2023-02-24T14:02:20Z
https://github.com/huggingface/datasets/issues/3053#issuecomment-1443724753
https://api.github.com/repos/huggingface/datasets/issues/3053
IC_kwDODunzps5WDX3R
https://api.github.com/repos/huggingface/datasets/issues/comments/1443724753/reactions
2023-02-24T14:02:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1443724753
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
## Describe the bug
When loading `the_pile_openwebtext2`, we get the error `pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type`

## Steps to reproduce the bug
```python
import datasets
ds = datasets.load_dataset('the_pile_openwebtext2')
```

## Expected results
Should download the dataset, convert it to an arrow file, and return a working Dataset object.

## Actual results
The download works, but conversion to the arrow file fails as follows:

```
>>> ds = datasets.load_dataset('the_pile_openwebtext2')
Downloading and preparing dataset openwebtext2/plain_text (download: 27.33 GiB, generated: 63.86 GiB
, post-processed: Unknown size, total: 91.19 GiB) to /home/davidbau/.cache/huggingface/datasets/open
webtext2/plain_text/1.0.0/c48ec73ba3483bac673463f48f67e9a4fd8cb49a9d6ec4fb957f0b424b97cf25...
Traceback (most recent call last):
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/builder.py", line 1133,
 in _prepare_split
    writer.write(example, key)
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
366, in write
    self.write_examples_on_file()
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
311, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
115, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
```

- Platform: Ubuntu 20.04
- Python version: python 3.9
- PyArrow version: 3.0.0
