ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'
MEMBER
Thanks for reporting. We are investigating it.
2023-08-31T09:33:33Z
https://github.com/huggingface/datasets/issues/6197#issuecomment-1700695823
https://api.github.com/repos/huggingface/datasets/issues/6197
IC_kwDODunzps5lXo8P
https://api.github.com/repos/huggingface/datasets/issues/comments/1700695823/reactions
2023-08-31T09:33:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1700695823
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
This issue is caused by latest `pandas` release 2.1.0 (released yesterday Aug 30).

See: https://github.com/huggingface/datasets/actions/runs/6035484010/job/16375932085?pr=6198

2023-08-31T09:39:04Z
https://github.com/huggingface/datasets/issues/6197#issuecomment-1700704708
https://api.github.com/repos/huggingface/datasets/issues/6197
IC_kwDODunzps5lXrHE
https://api.github.com/repos/huggingface/datasets/issues/comments/1700704708/reactions
2023-08-31T09:39:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1700704708
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
People using previous releases of `datasets` should pin `pandas` in their local environment:
```
python -m pip install 'pandas<2.1.0'
```
2023-09-01T10:34:40Z
https://github.com/huggingface/datasets/issues/6197#issuecomment-1702537569
https://api.github.com/repos/huggingface/datasets/issues/6197
IC_kwDODunzps5leqlh
https://api.github.com/repos/huggingface/datasets/issues/comments/1702537569/reactions
2023-09-01T10:35:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1702537569
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Describe the bug

Saving a dataset `.to_json()` fails with a `ValueError` since the latest `pandas` [release](https://pandas.pydata.org/docs/dev/whatsnew/v2.1.0.html) (`2.1.0`)

In their latest release we have:

> Improved error handling when using [DataFrame.to_json()](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json) with incompatible index and orient arguments ([GH 52143](https://github.com/pandas-dev/pandas/issues/52143))

i.e. an error is now raised for invalid combinations of `index` and `orient`.

This means that unfortunately the custom logic at this line might sometimes lead to contradictions:

https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/io/json.py#L96

e.g. for the default case `orient=records` leads to `index=True`, which now raises a `ValueError`

### Steps to reproduce the bug

```python
import datasets


if __name__ == '__main__':

    dataset = datasets.Dataset.from_dict({"A": [1, 2, 3], "B": [4, 5, 6]})
    dataset.to_json("dataset.json")
```

```shell
>>>
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'.
```

### Expected behavior

The dataset is successfully saved as `.json`

### Environment info

`python >= 3.9`
`pandas >= 2.1.0`