Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility
COLLABORATOR
Based on https://github.com/rapidsai/cudf/issues/10187, this probably means your `pyarrow` installation is not compatible with `datasets`.

Can you please execute the following commands in the terminal and paste the output here?
```
conda list | grep arrow
``` 
```
python -c "import pyarrow; print(pyarrow.__file__)"
```



2023-06-02T16:11:03Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1573979908
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d0QcE
https://api.github.com/repos/huggingface/datasets/issues/comments/1573979908/reactions
2023-06-02T16:11:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1573979908
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> Based on [rapidsai/cudf#10187](https://github.com/rapidsai/cudf/issues/10187), this probably means your `pyarrow` installation is not compatible with `datasets`.
> 
> Can you please execute the following commands in the terminal and paste the output here?
> 
> ```
> conda list | grep arrow
> ```
> 
> ```
> python -c "import pyarrow; print(pyarrow.__file__)"
> ```


Here is the output to the first command:
```
arrow-cpp                 11.0.0           py39h7f74497_0  
pyarrow                   12.0.0                   pypi_0    pypi
```
and the second:
```
/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/__init__.py
```
Thanks!




2023-06-02T19:47:30Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1574233233
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d1OSR
https://api.github.com/repos/huggingface/datasets/issues/comments/1574233233/reactions
2023-06-02T19:47:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1574233233
https://avatars.githubusercontent.com/u/71412682?v=4
https://api.github.com/users/ehuangc/events{/privacy}
https://api.github.com/users/ehuangc/followers
https://api.github.com/users/ehuangc/following{/other_user}
https://api.github.com/users/ehuangc/gists{/gist_id}
https://github.com/ehuangc
ehuangc
MDQ6VXNlcjcxNDEyNjgy
https://api.github.com/users/ehuangc/orgs
https://api.github.com/users/ehuangc/received_events
https://api.github.com/users/ehuangc/repos
https://api.github.com/users/ehuangc/starred{/owner}{/repo}
https://api.github.com/users/ehuangc/subscriptions
User
https://api.github.com/users/ehuangc
public
NONE
after installing pytesseract 0.3.10, I got the above error. FYI 
2023-06-03T04:28:57Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1574636248
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d2wrY
https://api.github.com/repos/huggingface/datasets/issues/comments/1574636248/reactions
2023-06-03T04:28:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1574636248
https://avatars.githubusercontent.com/u/57236540?v=4
https://api.github.com/users/Joheun-Kang/events{/privacy}
https://api.github.com/users/Joheun-Kang/followers
https://api.github.com/users/Joheun-Kang/following{/other_user}
https://api.github.com/users/Joheun-Kang/gists{/gist_id}
https://github.com/Joheun-Kang
Joheun-Kang
MDQ6VXNlcjU3MjM2NTQw
https://api.github.com/users/Joheun-Kang/orgs
https://api.github.com/users/Joheun-Kang/received_events
https://api.github.com/users/Joheun-Kang/repos
https://api.github.com/users/Joheun-Kang/starred{/owner}{/repo}
https://api.github.com/users/Joheun-Kang/subscriptions
User
https://api.github.com/users/Joheun-Kang
public
NONE
RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject
2023-06-03T04:45:14Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1574652229
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d20lF
https://api.github.com/repos/huggingface/datasets/issues/comments/1574652229/reactions
2023-06-03T04:45:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1574652229
https://avatars.githubusercontent.com/u/57236540?v=4
https://api.github.com/users/Joheun-Kang/events{/privacy}
https://api.github.com/users/Joheun-Kang/followers
https://api.github.com/users/Joheun-Kang/following{/other_user}
https://api.github.com/users/Joheun-Kang/gists{/gist_id}
https://github.com/Joheun-Kang
Joheun-Kang
MDQ6VXNlcjU3MjM2NTQw
https://api.github.com/users/Joheun-Kang/orgs
https://api.github.com/users/Joheun-Kang/received_events
https://api.github.com/users/Joheun-Kang/repos
https://api.github.com/users/Joheun-Kang/starred{/owner}{/repo}
https://api.github.com/users/Joheun-Kang/subscriptions
User
https://api.github.com/users/Joheun-Kang
public
NONE
I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.

Do we need to update dependencies? 
2023-06-04T23:32:19Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1575788697
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d7KCZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1575788697/reactions
2023-06-05T15:33:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1575788697
https://avatars.githubusercontent.com/u/6045564?v=4
https://api.github.com/users/ssydyc/events{/privacy}
https://api.github.com/users/ssydyc/followers
https://api.github.com/users/ssydyc/following{/other_user}
https://api.github.com/users/ssydyc/gists{/gist_id}
https://github.com/ssydyc
ssydyc
MDQ6VXNlcjYwNDU1NjQ=
https://api.github.com/users/ssydyc/orgs
https://api.github.com/users/ssydyc/received_events
https://api.github.com/users/ssydyc/repos
https://api.github.com/users/ssydyc/starred{/owner}{/repo}
https://api.github.com/users/ssydyc/subscriptions
User
https://api.github.com/users/ssydyc
public
MEMBER
Please note that our CI properly passes all tests with `pyarrow-12.0.0`, for Python 3.7 and Python 3.10, for Ubuntu and Windows: see for example https://github.com/huggingface/datasets/actions/runs/5157324334/jobs/9289582291
2023-06-05T05:31:55Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1576070619
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d8O3b
https://api.github.com/repos/huggingface/datasets/issues/comments/1576070619/reactions
2023-06-05T05:31:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1576070619
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
For conda with python3.8.16 this solved my problem! thanks!

> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies? I can work on that if no one else is working on it.


2023-06-05T07:41:21Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1576239034
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d83-6
https://api.github.com/repos/huggingface/datasets/issues/comments/1576239034/reactions
2023-06-05T07:43:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1576239034
https://avatars.githubusercontent.com/u/57236540?v=4
https://api.github.com/users/Joheun-Kang/events{/privacy}
https://api.github.com/users/Joheun-Kang/followers
https://api.github.com/users/Joheun-Kang/following{/other_user}
https://api.github.com/users/Joheun-Kang/gists{/gist_id}
https://github.com/Joheun-Kang
Joheun-Kang
MDQ6VXNlcjU3MjM2NTQw
https://api.github.com/users/Joheun-Kang/orgs
https://api.github.com/users/Joheun-Kang/received_events
https://api.github.com/users/Joheun-Kang/repos
https://api.github.com/users/Joheun-Kang/starred{/owner}{/repo}
https://api.github.com/users/Joheun-Kang/subscriptions
User
https://api.github.com/users/Joheun-Kang
public
NONE
Thanks for replying. I am not sure about those environments but it seems like pyarrow-12.0.0 does not work for conda with python 3.8.16. 

> Please note that our CI properly passes all tests with `pyarrow-12.0.0`, for Python 3.7 and Python 3.10, for Ubuntu and Windows: see for example https://github.com/huggingface/datasets/actions/runs/5157324334/jobs/9289582291


2023-06-05T07:42:29Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1576241508
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5d84lk
https://api.github.com/repos/huggingface/datasets/issues/comments/1576241508/reactions
2023-06-05T07:42:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1576241508
https://avatars.githubusercontent.com/u/57236540?v=4
https://api.github.com/users/Joheun-Kang/events{/privacy}
https://api.github.com/users/Joheun-Kang/followers
https://api.github.com/users/Joheun-Kang/following{/other_user}
https://api.github.com/users/Joheun-Kang/gists{/gist_id}
https://github.com/Joheun-Kang
Joheun-Kang
MDQ6VXNlcjU3MjM2NTQw
https://api.github.com/users/Joheun-Kang/orgs
https://api.github.com/users/Joheun-Kang/received_events
https://api.github.com/users/Joheun-Kang/repos
https://api.github.com/users/Joheun-Kang/starred{/owner}{/repo}
https://api.github.com/users/Joheun-Kang/subscriptions
User
https://api.github.com/users/Joheun-Kang
public
NONE
Got the same error with:

```
arrow-cpp                 11.0.0          py310h7516544_0  
pyarrow                   12.0.0                   pypi_0    pypi

python                    3.10.11              h7a1cb2a_2  

datasets                  2.13.0             pyhd8ed1ab_0    conda-forge
```
2023-06-19T10:10:13Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1596904395
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5fLtPL
https://api.github.com/repos/huggingface/datasets/issues/comments/1596904395/reactions
2023-06-19T10:10:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1596904395
https://avatars.githubusercontent.com/u/26467076?v=4
https://api.github.com/users/lorelupo/events{/privacy}
https://api.github.com/users/lorelupo/followers
https://api.github.com/users/lorelupo/following{/other_user}
https://api.github.com/users/lorelupo/gists{/gist_id}
https://github.com/lorelupo
lorelupo
MDQ6VXNlcjI2NDY3MDc2
https://api.github.com/users/lorelupo/orgs
https://api.github.com/users/lorelupo/received_events
https://api.github.com/users/lorelupo/repos
https://api.github.com/users/lorelupo/starred{/owner}{/repo}
https://api.github.com/users/lorelupo/subscriptions
User
https://api.github.com/users/lorelupo
public
NONE
> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies?

This solved the issue for me as well.
2023-06-19T10:16:04Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1596914519
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5fLvtX
https://api.github.com/repos/huggingface/datasets/issues/comments/1596914519/reactions
2023-06-19T10:16:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1596914519
https://avatars.githubusercontent.com/u/26467076?v=4
https://api.github.com/users/lorelupo/events{/privacy}
https://api.github.com/users/lorelupo/followers
https://api.github.com/users/lorelupo/following{/other_user}
https://api.github.com/users/lorelupo/gists{/gist_id}
https://github.com/lorelupo
lorelupo
MDQ6VXNlcjI2NDY3MDc2
https://api.github.com/users/lorelupo/orgs
https://api.github.com/users/lorelupo/received_events
https://api.github.com/users/lorelupo/repos
https://api.github.com/users/lorelupo/starred{/owner}{/repo}
https://api.github.com/users/lorelupo/subscriptions
User
https://api.github.com/users/lorelupo
public
NONE
> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies?

Solved it for me also
2023-06-26T17:14:49Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1607895634
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5f1opS
https://api.github.com/repos/huggingface/datasets/issues/comments/1607895634/reactions
2023-06-26T17:14:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1607895634
https://avatars.githubusercontent.com/u/102803372?v=4
https://api.github.com/users/imarquart/events{/privacy}
https://api.github.com/users/imarquart/followers
https://api.github.com/users/imarquart/following{/other_user}
https://api.github.com/users/imarquart/gists{/gist_id}
https://github.com/imarquart
imarquart
U_kgDOBiCnrA
https://api.github.com/users/imarquart/orgs
https://api.github.com/users/imarquart/received_events
https://api.github.com/users/imarquart/repos
https://api.github.com/users/imarquart/starred{/owner}{/repo}
https://api.github.com/users/imarquart/subscriptions
User
https://api.github.com/users/imarquart
public
NONE
> 基于 [rapidsai/cudf#10187](https://github.com/rapidsai/cudf/issues/10187)，这可能意味着您的安装与 不兼容。`pyarrow``datasets`
> 
> 您能否在终端中执行以下命令并将输出粘贴到此处？
> 
> ```
> conda list | grep arrow
> ```
> 
> ```
> python -c "import pyarrow; print(pyarrow.__file__)"
> ```

arrow-cpp                 11.0.0          py310h7516544_0  
pyarrow                   12.0.1                   pypi_0    pypi

/root/miniconda3/lib/python3.10/site-packages/pyarrow/__init__.py
2023-07-11T11:37:17Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1630664108
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5hMfWs
https://api.github.com/repos/huggingface/datasets/issues/comments/1630664108/reactions
2023-07-11T11:37:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1630664108
https://avatars.githubusercontent.com/u/91943053?v=4
https://api.github.com/users/YY0649/events{/privacy}
https://api.github.com/users/YY0649/followers
https://api.github.com/users/YY0649/following{/other_user}
https://api.github.com/users/YY0649/gists{/gist_id}
https://github.com/YY0649
YY0649
U_kgDOBXrwjQ
https://api.github.com/users/YY0649/orgs
https://api.github.com/users/YY0649/received_events
https://api.github.com/users/YY0649/repos
https://api.github.com/users/YY0649/starred{/owner}{/repo}
https://api.github.com/users/YY0649/subscriptions
User
https://api.github.com/users/YY0649
public
NONE
Got the same problem with

arrow-cpp                 11.0.0          py310h1fc3239_0  
pyarrow                   12.0.1                   pypi_0    pypi

miniforge3/envs/mlp/lib/python3.10/site-packages/pyarrow/__init__.py

Reverting back to pyarrow 11 solved the problem.

2023-07-23T20:39:59Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1646954716
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5iKojc
https://api.github.com/repos/huggingface/datasets/issues/comments/1646954716/reactions
2023-07-23T20:39:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1646954716
https://avatars.githubusercontent.com/u/35255751?v=4
https://api.github.com/users/kimjansheden/events{/privacy}
https://api.github.com/users/kimjansheden/followers
https://api.github.com/users/kimjansheden/following{/other_user}
https://api.github.com/users/kimjansheden/gists{/gist_id}
https://github.com/kimjansheden
kimjansheden
MDQ6VXNlcjM1MjU1NzUx
https://api.github.com/users/kimjansheden/orgs
https://api.github.com/users/kimjansheden/received_events
https://api.github.com/users/kimjansheden/repos
https://api.github.com/users/kimjansheden/starred{/owner}{/repo}
https://api.github.com/users/kimjansheden/subscriptions
User
https://api.github.com/users/kimjansheden
public
NONE
Solved with `pip install pyarrow==11.0.0`
2023-08-07T08:59:33Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1667471701
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5jY5lV
https://api.github.com/repos/huggingface/datasets/issues/comments/1667471701/reactions
2023-08-07T08:59:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1667471701
https://avatars.githubusercontent.com/u/100468506?v=4
https://api.github.com/users/B8ni/events{/privacy}
https://api.github.com/users/B8ni/followers
https://api.github.com/users/B8ni/following{/other_user}
https://api.github.com/users/B8ni/gists{/gist_id}
https://github.com/B8ni
B8ni
U_kgDOBf0HGg
https://api.github.com/users/B8ni/orgs
https://api.github.com/users/B8ni/received_events
https://api.github.com/users/B8ni/repos
https://api.github.com/users/B8ni/starred{/owner}{/repo}
https://api.github.com/users/B8ni/subscriptions
User
https://api.github.com/users/B8ni
public
NONE
I got different. Solved with
pip install pyarrow==12.0.1
pip install cchardet

env:
Python 3.9.16
transformers  4.32.1
2023-08-31T02:02:24Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1700251233
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5lV8Zh
https://api.github.com/repos/huggingface/datasets/issues/comments/1700251233/reactions
2023-08-31T02:02:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1700251233
https://avatars.githubusercontent.com/u/59823739?v=4
https://api.github.com/users/wolf-li/events{/privacy}
https://api.github.com/users/wolf-li/followers
https://api.github.com/users/wolf-li/following{/other_user}
https://api.github.com/users/wolf-li/gists{/gist_id}
https://github.com/wolf-li
wolf-li
MDQ6VXNlcjU5ODIzNzM5
https://api.github.com/users/wolf-li/orgs
https://api.github.com/users/wolf-li/received_events
https://api.github.com/users/wolf-li/repos
https://api.github.com/users/wolf-li/starred{/owner}{/repo}
https://api.github.com/users/wolf-li/subscriptions
User
https://api.github.com/users/wolf-li
public
NONE
> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies?

This works for me as well
2023-09-03T04:56:31Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1704006889
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5lkRTp
https://api.github.com/repos/huggingface/datasets/issues/comments/1704006889/reactions
2023-09-03T04:56:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1704006889
https://avatars.githubusercontent.com/u/95742992?v=4
https://api.github.com/users/5uryansh/events{/privacy}
https://api.github.com/users/5uryansh/followers
https://api.github.com/users/5uryansh/following{/other_user}
https://api.github.com/users/5uryansh/gists{/gist_id}
https://github.com/5uryansh
5uryansh
U_kgDOBbTsEA
https://api.github.com/users/5uryansh/orgs
https://api.github.com/users/5uryansh/received_events
https://api.github.com/users/5uryansh/repos
https://api.github.com/users/5uryansh/starred{/owner}{/repo}
https://api.github.com/users/5uryansh/subscriptions
User
https://api.github.com/users/5uryansh
public
NONE
> I got different. Solved with pip install pyarrow==12.0.1 pip install cchardet
> 
> env: Python 3.9.16 transformers 4.32.1

I guess it also depends on the Python version. I got Python 3.11.5 and pyarrow==12.0.0. 
It works! 
2023-11-08T20:20:56Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1802593065
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5rcWMp
https://api.github.com/repos/huggingface/datasets/issues/comments/1802593065/reactions
2023-11-08T20:20:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1802593065
https://avatars.githubusercontent.com/u/20698803?v=4
https://api.github.com/users/williamLyh/events{/privacy}
https://api.github.com/users/williamLyh/followers
https://api.github.com/users/williamLyh/following{/other_user}
https://api.github.com/users/williamLyh/gists{/gist_id}
https://github.com/williamLyh
williamLyh
MDQ6VXNlcjIwNjk4ODAz
https://api.github.com/users/williamLyh/orgs
https://api.github.com/users/williamLyh/received_events
https://api.github.com/users/williamLyh/repos
https://api.github.com/users/williamLyh/starred{/owner}{/repo}
https://api.github.com/users/williamLyh/subscriptions
User
https://api.github.com/users/williamLyh
public
NONE
Hi, if this helps anyone, pip install pyarrow==11.0.0 did not work for me (I'm using Colab) but this worked: 
!pip install --extra-index-url=https://pypi.nvidia.com cudf-cu11
2023-12-13T15:53:51Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1854191628
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5uhLgM
https://api.github.com/repos/huggingface/datasets/issues/comments/1854191628/reactions
2023-12-13T15:53:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1854191628
https://avatars.githubusercontent.com/u/13047683?v=4
https://api.github.com/users/thierrydecae/events{/privacy}
https://api.github.com/users/thierrydecae/followers
https://api.github.com/users/thierrydecae/following{/other_user}
https://api.github.com/users/thierrydecae/gists{/gist_id}
https://github.com/thierrydecae
thierrydecae
MDQ6VXNlcjEzMDQ3Njgz
https://api.github.com/users/thierrydecae/orgs
https://api.github.com/users/thierrydecae/received_events
https://api.github.com/users/thierrydecae/repos
https://api.github.com/users/thierrydecae/starred{/owner}{/repo}
https://api.github.com/users/thierrydecae/subscriptions
User
https://api.github.com/users/thierrydecae
public
NONE
> Hi, if this helps anyone, pip install pyarrow==11.0.0 did not work for me (I'm using Colab) but this worked: !pip install --extra-index-url=https://pypi.nvidia.com cudf-cu11

thanks! I met the same problem and your suggestion solved it.
2024-02-07T17:47:36Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1932569364
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5zMKsU
https://api.github.com/repos/huggingface/datasets/issues/comments/1932569364/reactions
2024-02-07T17:47:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1932569364
https://avatars.githubusercontent.com/u/105380124?v=4
https://api.github.com/users/JerryRen471/events{/privacy}
https://api.github.com/users/JerryRen471/followers
https://api.github.com/users/JerryRen471/following{/other_user}
https://api.github.com/users/JerryRen471/gists{/gist_id}
https://github.com/JerryRen471
JerryRen471
U_kgDOBkf5HA
https://api.github.com/users/JerryRen471/orgs
https://api.github.com/users/JerryRen471/received_events
https://api.github.com/users/JerryRen471/repos
https://api.github.com/users/JerryRen471/starred{/owner}{/repo}
https://api.github.com/users/JerryRen471/subscriptions
User
https://api.github.com/users/JerryRen471
public
CONTRIBUTOR
(I was doing quiet install so I didn't notice it initially)
I've been loading the same dataset for months on Colab, just now I got this error as well. I think Colab has changed their image recently (I had some errors regarding CUDA previously as well). beware of this and restart runtime if you're doing quite pip installs.
moreover installing stable version of datasets on pypi gives this:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ibis-framework 7.1.0 requires pyarrow<15,>=2, but you have pyarrow 15.0.0 which is incompatible.
Successfully installed datasets-2.17.0 dill-0.3.8 multiprocess-0.70.16 pyarrow-15.0.0
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
``` 

2024-02-10T22:54:58Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1937343296
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5zeYNA
https://api.github.com/repos/huggingface/datasets/issues/comments/1937343296/reactions
2024-02-10T22:58:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1937343296
https://avatars.githubusercontent.com/u/53175384?v=4
https://api.github.com/users/merveenoyan/events{/privacy}
https://api.github.com/users/merveenoyan/followers
https://api.github.com/users/merveenoyan/following{/other_user}
https://api.github.com/users/merveenoyan/gists{/gist_id}
https://github.com/merveenoyan
merveenoyan
MDQ6VXNlcjUzMTc1Mzg0
https://api.github.com/users/merveenoyan/orgs
https://api.github.com/users/merveenoyan/received_events
https://api.github.com/users/merveenoyan/repos
https://api.github.com/users/merveenoyan/starred{/owner}{/repo}
https://api.github.com/users/merveenoyan/subscriptions
User
https://api.github.com/users/merveenoyan
public
NONE
for colab - pip install pyarrow==11.0.0
2024-02-15T01:46:13Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1945233690
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5z8eka
https://api.github.com/repos/huggingface/datasets/issues/comments/1945233690/reactions
2024-02-15T01:46:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1945233690
https://avatars.githubusercontent.com/u/77905051?v=4
https://api.github.com/users/rasith1998/events{/privacy}
https://api.github.com/users/rasith1998/followers
https://api.github.com/users/rasith1998/following{/other_user}
https://api.github.com/users/rasith1998/gists{/gist_id}
https://github.com/rasith1998
rasith1998
MDQ6VXNlcjc3OTA1MDUx
https://api.github.com/users/rasith1998/orgs
https://api.github.com/users/rasith1998/received_events
https://api.github.com/users/rasith1998/repos
https://api.github.com/users/rasith1998/starred{/owner}{/repo}
https://api.github.com/users/rasith1998/subscriptions
User
https://api.github.com/users/rasith1998
public
NONE
The above methods didn't help me. So I installed an older version: `!pip install datasets==2.16.1`
and `import datasets` worked!!
2024-02-15T06:52:56Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1945466888
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps5z9XgI
https://api.github.com/repos/huggingface/datasets/issues/comments/1945466888/reactions
2024-02-15T06:52:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1945466888
https://avatars.githubusercontent.com/u/123853388?v=4
https://api.github.com/users/PennlaineChu/events{/privacy}
https://api.github.com/users/PennlaineChu/followers
https://api.github.com/users/PennlaineChu/following{/other_user}
https://api.github.com/users/PennlaineChu/gists{/gist_id}
https://github.com/PennlaineChu
PennlaineChu
U_kgDOB2HaTA
https://api.github.com/users/PennlaineChu/orgs
https://api.github.com/users/PennlaineChu/received_events
https://api.github.com/users/PennlaineChu/repos
https://api.github.com/users/PennlaineChu/starred{/owner}{/repo}
https://api.github.com/users/PennlaineChu/subscriptions
User
https://api.github.com/users/PennlaineChu
public
COLLABORATOR
@rasith1998 @PennlaineChu You can avoid this issue by restarting the session after the `datasets` installation (see https://github.com/huggingface/datasets/issues/6661 for more info)

Also, we've contacted Google Colab folks to update the default PyArrow installation, so the issue should soon be "officially" resolved on their side.
2024-02-15T17:51:36Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1946760462
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps50CTUO
https://api.github.com/repos/huggingface/datasets/issues/comments/1946760462/reactions
2024-02-15T17:51:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1946760462
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
COLLABORATOR
> Also, we've contacted Google Colab folks to update the default PyArrow installation, so the issue should soon be "officially" resolved on their side.

This has been done! Google Colab now pre-installs PyArrow 14.0.2, which makes this issue unlikely to happen, so I'm closing it.
2024-02-25T16:37:40Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-1962994058
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps51AOmK
https://api.github.com/repos/huggingface/datasets/issues/comments/1962994058/reactions
2024-02-25T16:37:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1962994058
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I am facing this issue outside of Colab, in a normal Python (3.10.14) environment:
```
pyarrow==11.0.0
datasets=2.20.0
transformers==4.41.2
```

What can I do to solve it?

I am somewhat bound to `pyarrow==11.0.0`. Is there a version of `datasets` that supports this?
2024-06-27T09:13:17Z
https://github.com/huggingface/datasets/issues/5923#issuecomment-2194184055
https://api.github.com/repos/huggingface/datasets/issues/5923
IC_kwDODunzps6CyJd3
https://api.github.com/repos/huggingface/datasets/issues/comments/2194184055/reactions
2024-06-27T10:07:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2194184055
https://avatars.githubusercontent.com/u/49385643?v=4
https://api.github.com/users/MinuraPunchihewa/events{/privacy}
https://api.github.com/users/MinuraPunchihewa/followers
https://api.github.com/users/MinuraPunchihewa/following{/other_user}
https://api.github.com/users/MinuraPunchihewa/gists{/gist_id}
https://github.com/MinuraPunchihewa
MinuraPunchihewa
MDQ6VXNlcjQ5Mzg1NjQz
https://api.github.com/users/MinuraPunchihewa/orgs
https://api.github.com/users/MinuraPunchihewa/received_events
https://api.github.com/users/MinuraPunchihewa/repos
https://api.github.com/users/MinuraPunchihewa/starred{/owner}{/repo}
https://api.github.com/users/MinuraPunchihewa/subscriptions
User
https://api.github.com/users/MinuraPunchihewa
public
### Describe the bug

When trying to import datasets, I get a pyarrow ValueError:

Traceback (most recent call last):
  File "/Users/edward/test/test.py", line 1, in <module>
    import datasets
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 65, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_reader.py", line 28, in <module>
    import pyarrow.parquet as pq
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 20, in <module>
    from .core import *
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 45, in <module>
    from pyarrow.fs import (LocalFileSystem, FileSystem, FileType,
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/fs.py", line 49, in <module>
    from pyarrow._gcsfs import GcsFileSystem  # noqa
  File "pyarrow/_gcsfs.pyx", line 1, in init pyarrow._gcsfs
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

`import datasets`

### Expected behavior

Successful import

### Environment info

Conda environment, MacOS
python 3.9.12
datasets 2.12.0

