ImportError: cannot import name 'SchemaInferenceError' from 'datasets.arrow_writer' (/opt/conda/lib/python3.10/site-packages/datasets/arrow_writer.py)
MEMBER
Hi ! Are you sure you have `datasets` 2.16 ? I just checked and on 2.16 I can run `from datasets.arrow_writer import SchemaInferenceError` without error
2023-12-30T16:14:11Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1872557464
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vnPWY
https://api.github.com/repos/huggingface/datasets/issues/comments/1872557464/reactions
2023-12-30T16:14:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1872557464
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I have the same issue - using with datasets version 2.16.1. Also this is on a kaggle notebook - other people with the same issue also seem to be having it on kaggle?
2023-12-30T19:58:04Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1872593912
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vnYP4
https://api.github.com/repos/huggingface/datasets/issues/comments/1872593912/reactions
2023-12-30T19:58:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1872593912
https://avatars.githubusercontent.com/u/100781655?v=4
https://api.github.com/users/iwo9/events{/privacy}
https://api.github.com/users/iwo9/followers
https://api.github.com/users/iwo9/following{/other_user}
https://api.github.com/users/iwo9/gists{/gist_id}
https://github.com/iwo9
iwo9
U_kgDOBgHOVw
https://api.github.com/users/iwo9/orgs
https://api.github.com/users/iwo9/received_events
https://api.github.com/users/iwo9/repos
https://api.github.com/users/iwo9/starred{/owner}{/repo}
https://api.github.com/users/iwo9/subscriptions
User
https://api.github.com/users/iwo9
public
NONE
I have the same issue now and didn't have this problem around 2 weeks ago.
2024-01-02T03:02:46Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1873585096
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vrKPI
https://api.github.com/repos/huggingface/datasets/issues/comments/1873585096/reactions
2024-01-02T03:02:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1873585096
https://avatars.githubusercontent.com/u/94343019?v=4
https://api.github.com/users/YZx0pa/events{/privacy}
https://api.github.com/users/YZx0pa/followers
https://api.github.com/users/YZx0pa/following{/other_user}
https://api.github.com/users/YZx0pa/gists{/gist_id}
https://github.com/YZx0pa
YZx0pa
U_kgDOBZ-Paw
https://api.github.com/users/YZx0pa/orgs
https://api.github.com/users/YZx0pa/received_events
https://api.github.com/users/YZx0pa/repos
https://api.github.com/users/YZx0pa/starred{/owner}{/repo}
https://api.github.com/users/YZx0pa/subscriptions
User
https://api.github.com/users/YZx0pa
public
NONE
> Hi ! Are you sure you have `datasets` 2.16 ? I just checked and on 2.16 I can run `from datasets.arrow_writer import SchemaInferenceError` without error

Yes, I am sure

```
!pip show datasets
Name: datasets
Version: 2.16.1
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /opt/conda/lib/python3.10/site-packages
Requires: aiohttp, dill, filelock, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, pyarrow-hotfix, pyyaml, requests, tqdm, xxhash
Required-by: trl
```
2024-01-02T04:41:26Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1873615686
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vrRtG
https://api.github.com/repos/huggingface/datasets/issues/comments/1873615686/reactions
2024-01-02T04:41:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1873615686
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
NONE
> I have the same issue - using with datasets version 2.16.1. Also this is on a kaggle notebook - other people with the same issue also seem to be having it on kaggle?

Don't know about other people. But I am having this issue whose solution I can't find anywhere. And this issue still persists. 
2024-01-02T04:44:29Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1873616620
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vrR7s
https://api.github.com/repos/huggingface/datasets/issues/comments/1873616620/reactions
2024-01-02T04:44:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1873616620
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
NONE
> I have the same issue now and didn't have this problem around 2 weeks ago.

Same here
2024-01-02T04:46:14Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1873617144
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vrSD4
https://api.github.com/repos/huggingface/datasets/issues/comments/1873617144/reactions
2024-01-02T04:46:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1873617144
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
NONE
I was having the same issue but the datasets version was 2.6.1, after I updated it to latest(2.16), error is gone while importing.

2024-01-02T11:49:38Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1873930120
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vseeI
https://api.github.com/repos/huggingface/datasets/issues/comments/1873930120/reactions
2024-01-02T11:49:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1873930120
https://avatars.githubusercontent.com/u/33358596?v=4
https://api.github.com/users/lucken99/events{/privacy}
https://api.github.com/users/lucken99/followers
https://api.github.com/users/lucken99/following{/other_user}
https://api.github.com/users/lucken99/gists{/gist_id}
https://github.com/lucken99
lucken99
MDQ6VXNlcjMzMzU4NTk2
https://api.github.com/users/lucken99/orgs
https://api.github.com/users/lucken99/received_events
https://api.github.com/users/lucken99/repos
https://api.github.com/users/lucken99/starred{/owner}{/repo}
https://api.github.com/users/lucken99/subscriptions
User
https://api.github.com/users/lucken99
public
NONE
> I was having the same issue but the datasets version was 2.6.1, after I updated it to latest(2.16), error is gone while importing.

I also have datasets version 2.16, but the error is still there.
2024-01-02T12:15:46Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1873955049
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vskjp
https://api.github.com/repos/huggingface/datasets/issues/comments/1873955049/reactions
2024-01-02T12:15:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1873955049
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
MEMBER
Can you try re-installing `datasets` ?
2024-01-02T13:48:04Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1874047879
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vs7OH
https://api.github.com/repos/huggingface/datasets/issues/comments/1874047879/reactions
2024-01-02T13:48:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874047879
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Can you try re-installing `datasets` ?

I tried re-installing. Still getting the same error. 

2024-01-03T04:43:25Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1874842727
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vv9Rn
https://api.github.com/repos/huggingface/datasets/issues/comments/1874842727/reactions
2024-01-03T04:43:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874842727
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
NONE
> > Can you try re-installing `datasets` ?
> 
> I tried re-installing. Still getting the same error.

In kaggle I used:
- `%pip install -U datasets`
and then restarted runtime and then everything works fine.
2024-01-03T06:53:03Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1874922883
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vwQ2D
https://api.github.com/repos/huggingface/datasets/issues/comments/1874922883/reactions
2024-01-03T06:53:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874922883
https://avatars.githubusercontent.com/u/33358596?v=4
https://api.github.com/users/lucken99/events{/privacy}
https://api.github.com/users/lucken99/followers
https://api.github.com/users/lucken99/following{/other_user}
https://api.github.com/users/lucken99/gists{/gist_id}
https://github.com/lucken99
lucken99
MDQ6VXNlcjMzMzU4NTk2
https://api.github.com/users/lucken99/orgs
https://api.github.com/users/lucken99/received_events
https://api.github.com/users/lucken99/repos
https://api.github.com/users/lucken99/starred{/owner}{/repo}
https://api.github.com/users/lucken99/subscriptions
User
https://api.github.com/users/lucken99
public
NONE
> > > Can you try re-installing `datasets` ?
> > 
> > 
> > I tried re-installing. Still getting the same error.
> 
> In kaggle I used:
> 
> * `%pip install -U datasets`
>   and then restarted runtime and then everything works fine.

Yes, this is working. When I restart the runtime after installing packages, it's working perfectly. Thank you so much. But why do we need to restart runtime every time after installing packages?
2024-01-03T09:36:35Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1875079884
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vw3LM
https://api.github.com/repos/huggingface/datasets/issues/comments/1875079884/reactions
2024-01-03T09:49:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1875079884
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
NONE
> > > > Can you try re-installing `datasets` ?
> > > 
> > > 
> > > I tried re-installing. Still getting the same error.
> > 
> > 
> > In kaggle I used:
> > 
> > * `%pip install -U datasets`
> >   and then restarted runtime and then everything works fine.
> 
> Yes, this is working. When I restart the runtime after installing packages, it's working perfectly. Thank you so much. But why do we need to restart runtime every time after installing packages?
For some packages it is required.
https://stackoverflow.com/questions/57831187/need-to-restart-runtime-before-import-an-installed-package-in-colab

2024-01-03T09:53:59Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1875102193
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vw8nx
https://api.github.com/repos/huggingface/datasets/issues/comments/1875102193/reactions
2024-01-03T09:54:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1875102193
https://avatars.githubusercontent.com/u/33358596?v=4
https://api.github.com/users/lucken99/events{/privacy}
https://api.github.com/users/lucken99/followers
https://api.github.com/users/lucken99/following{/other_user}
https://api.github.com/users/lucken99/gists{/gist_id}
https://github.com/lucken99
lucken99
MDQ6VXNlcjMzMzU4NTk2
https://api.github.com/users/lucken99/orgs
https://api.github.com/users/lucken99/received_events
https://api.github.com/users/lucken99/repos
https://api.github.com/users/lucken99/starred{/owner}{/repo}
https://api.github.com/users/lucken99/subscriptions
User
https://api.github.com/users/lucken99
public
NONE
> > > > > Can you try re-installing `datasets` ?
> > > > 
> > > > 
> > > > I tried re-installing. Still getting the same error.
> > > 
> > > 
> > > In kaggle I used:
> > > 
> > > * `%pip install -U datasets`
> > >   and then restarted runtime and then everything works fine.
> > 
> > 
> > Yes, this is working. When I restart the runtime after installing packages, it's working perfectly. Thank you so much. But why do we need to restart runtime every time after installing packages?
> > For some packages it is required.
> > https://stackoverflow.com/questions/57831187/need-to-restart-runtime-before-import-an-installed-package-in-colab

Thank you for your assistance. I dedicated the past 2-3 weeks to resolving this issue. Interestingly, it runs flawlessly in Colab without requiring a runtime restart. However, the problem persisted exclusively in Kaggle. I appreciate your help once again. Thank you.
2024-01-03T10:03:44Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1875114121
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vw_iJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1875114121/reactions
2024-01-03T10:03:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1875114121
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
NONE
Closing this issue as it is not related to the datasets library; rather, it's linked to platform-related issues.
2024-01-03T10:06:46Z
https://github.com/huggingface/datasets/issues/6538#issuecomment-1875118343
https://api.github.com/repos/huggingface/datasets/issues/6538
IC_kwDODunzps5vxAkH
https://api.github.com/repos/huggingface/datasets/issues/comments/1875118343/reactions
2024-01-03T10:06:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1875118343
https://avatars.githubusercontent.com/u/131662185?v=4
https://api.github.com/users/Sonali-Behera-TRT/events{/privacy}
https://api.github.com/users/Sonali-Behera-TRT/followers
https://api.github.com/users/Sonali-Behera-TRT/following{/other_user}
https://api.github.com/users/Sonali-Behera-TRT/gists{/gist_id}
https://github.com/Sonali-Behera-TRT
Sonali-Behera-TRT
U_kgDOB9kBaQ
https://api.github.com/users/Sonali-Behera-TRT/orgs
https://api.github.com/users/Sonali-Behera-TRT/received_events
https://api.github.com/users/Sonali-Behera-TRT/repos
https://api.github.com/users/Sonali-Behera-TRT/starred{/owner}{/repo}
https://api.github.com/users/Sonali-Behera-TRT/subscriptions
User
https://api.github.com/users/Sonali-Behera-TRT
public
### Describe the bug

While importing from packages getting the error
Code:

```
import os
import torch
from datasets import load_dataset, Dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging
)
from peft import LoraConfig, PeftModel
from trl import SFTTrainer
from huggingface_hub import login
import pandas as pd
```

Error:
````
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[5], line 14
      4 from transformers import (
      5     AutoModelForCausalLM,
      6     AutoTokenizer,
   (...)
     11     logging
     12 )
     13 from peft import LoraConfig, PeftModel
---> 14 from trl import SFTTrainer
     15 from huggingface_hub import login
     16 import pandas as pd

File /opt/conda/lib/python3.10/site-packages/trl/__init__.py:21
      8 from .import_utils import (
      9     is_diffusers_available,
     10     is_npu_available,
   (...)
     13     is_xpu_available,
     14 )
     15 from .models import (
     16     AutoModelForCausalLMWithValueHead,
     17     AutoModelForSeq2SeqLMWithValueHead,
     18     PreTrainedModelWrapper,
     19     create_reference_model,
     20 )
---> 21 from .trainer import (
     22     DataCollatorForCompletionOnlyLM,
     23     DPOTrainer,
     24     IterativeSFTTrainer,
     25     PPOConfig,
     26     PPOTrainer,
     27     RewardConfig,
     28     RewardTrainer,
     29     SFTTrainer,
     30 )
     33 if is_diffusers_available():
     34     from .models import (
     35         DDPOPipelineOutput,
     36         DDPOSchedulerOutput,
     37         DDPOStableDiffusionPipeline,
     38         DefaultDDPOStableDiffusionPipeline,
     39     )

File /opt/conda/lib/python3.10/site-packages/trl/trainer/__init__.py:44
     42 from .ppo_trainer import PPOTrainer
     43 from .reward_trainer import RewardTrainer, compute_accuracy
---> 44 from .sft_trainer import SFTTrainer
     45 from .training_configs import RewardConfig

File /opt/conda/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:23
     21 import torch.nn as nn
     22 from datasets import Dataset
---> 23 from datasets.arrow_writer import SchemaInferenceError
     24 from datasets.builder import DatasetGenerationError
     25 from transformers import (
     26     AutoModelForCausalLM,
     27     AutoTokenizer,
   (...)
     33     TrainingArguments,
     34 )

ImportError: cannot import name 'SchemaInferenceError' from 'datasets.arrow_writer' (/opt/conda/lib/python3.10/site-packages/datasets/arrow_writer.py

````

transformers version: 4.36.2
python version: 3.10.12
datasets version: 2.16.1

### Steps to reproduce the bug

1. Install packages
```
!pip install -U datasets trl accelerate peft bitsandbytes transformers trl huggingface_hub
```

2. import packages
```
import os
import torch
from datasets import load_dataset, Dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging
)
from peft import LoraConfig, PeftModel
from trl import SFTTrainer
from huggingface_hub import login
import pandas as pd
```

### Expected behavior

No error while importing

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.15.133+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 11.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0