Problems after upgrading to 2.6.1
COLLABORATOR
Hi! I can't reproduce the error following these steps. Can you please provide a reproducible example?
2022-10-24T18:39:58Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1289442338
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5M21Qi
https://api.github.com/repos/huggingface/datasets/issues/comments/1289442338/reactions
2022-10-24T18:39:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1289442338
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I faced the same issue:

### Repro
```
!pip install datasets==2.6.1
import datasets as Dataset
dataset = Dataset.from_pandas(dataframe)
dataset.save_to_disk(local)

!pip install datasets==2.5.2
import datasets as Dataset
dataset = Dataset.load_from_disk(local)
```


2022-10-24T19:12:15Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1289478571
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5M2-Gr
https://api.github.com/repos/huggingface/datasets/issues/comments/1289478571/reactions
2022-10-24T19:12:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1289478571
https://avatars.githubusercontent.com/u/36520926?v=4
https://api.github.com/users/Lokiiiiii/events{/privacy}
https://api.github.com/users/Lokiiiiii/followers
https://api.github.com/users/Lokiiiiii/following{/other_user}
https://api.github.com/users/Lokiiiiii/gists{/gist_id}
https://github.com/Lokiiiiii
Lokiiiiii
MDQ6VXNlcjM2NTIwOTI2
https://api.github.com/users/Lokiiiiii/orgs
https://api.github.com/users/Lokiiiiii/received_events
https://api.github.com/users/Lokiiiiii/repos
https://api.github.com/users/Lokiiiiii/starred{/owner}{/repo}
https://api.github.com/users/Lokiiiiii/subscriptions
User
https://api.github.com/users/Lokiiiiii
public
COLLABORATOR
@Lokiiiiii And what are the contents of the "dataframe" in your example?
2022-10-25T12:10:32Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1290444666
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5M6p96
https://api.github.com/repos/huggingface/datasets/issues/comments/1290444666/reactions
2022-10-25T12:10:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1290444666
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I bumped into the issue too. @Lokiiiiii thanks for steps. I "solved" if for now by `pip install datasets>=2.6.1` everywhere.
2022-10-26T16:33:32Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1292309822
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5NBxU-
https://api.github.com/repos/huggingface/datasets/issues/comments/1292309822/reactions
2022-10-26T16:33:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1292309822
https://avatars.githubusercontent.com/u/22592860?v=4
https://api.github.com/users/vvalouch/events{/privacy}
https://api.github.com/users/vvalouch/followers
https://api.github.com/users/vvalouch/following{/other_user}
https://api.github.com/users/vvalouch/gists{/gist_id}
https://github.com/vvalouch
vvalouch
MDQ6VXNlcjIyNTkyODYw
https://api.github.com/users/vvalouch/orgs
https://api.github.com/users/vvalouch/received_events
https://api.github.com/users/vvalouch/repos
https://api.github.com/users/vvalouch/starred{/owner}{/repo}
https://api.github.com/users/vvalouch/subscriptions
User
https://api.github.com/users/vvalouch
public
NONE
Hi all, 
I experienced the same issue. 
Please note that the pull request is related to the IMDB example provided in the doc, and is a fix for that, in that context, to make sure that people can follow the doc example and have a working system. 
It does not provide a fix for Datasets itself. 
2022-11-18T10:37:56Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1319823807
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5Oqum_
https://api.github.com/repos/huggingface/datasets/issues/comments/1319823807/reactions
2022-11-18T10:37:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1319823807
https://avatars.githubusercontent.com/u/23473052?v=4
https://api.github.com/users/maxpastor/events{/privacy}
https://api.github.com/users/maxpastor/followers
https://api.github.com/users/maxpastor/following{/other_user}
https://api.github.com/users/maxpastor/gists{/gist_id}
https://github.com/maxpastor
maxpastor
MDQ6VXNlcjIzNDczMDUy
https://api.github.com/users/maxpastor/orgs
https://api.github.com/users/maxpastor/received_events
https://api.github.com/users/maxpastor/repos
https://api.github.com/users/maxpastor/starred{/owner}{/repo}
https://api.github.com/users/maxpastor/subscriptions
User
https://api.github.com/users/maxpastor
public
NONE
im getting the same error.
- using the base AWS HF container that uses a datasets <2.
- updating the AWS HF container to use dataset 2.4

2022-12-08T20:26:20Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1343311970
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5QEVBi
https://api.github.com/repos/huggingface/datasets/issues/comments/1343311970/reactions
2022-12-08T20:26:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1343311970
https://avatars.githubusercontent.com/u/40301721?v=4
https://api.github.com/users/d-v-dlee/events{/privacy}
https://api.github.com/users/d-v-dlee/followers
https://api.github.com/users/d-v-dlee/following{/other_user}
https://api.github.com/users/d-v-dlee/gists{/gist_id}
https://github.com/d-v-dlee
d-v-dlee
MDQ6VXNlcjQwMzAxNzIx
https://api.github.com/users/d-v-dlee/orgs
https://api.github.com/users/d-v-dlee/received_events
https://api.github.com/users/d-v-dlee/repos
https://api.github.com/users/d-v-dlee/starred{/owner}{/repo}
https://api.github.com/users/d-v-dlee/subscriptions
User
https://api.github.com/users/d-v-dlee
public
NONE
Same here, running on our SageMaker pipelines. It's only happening for some but not all of our saved Datasets.
2022-12-16T15:59:54Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1355137064
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5QxcAo
https://api.github.com/repos/huggingface/datasets/issues/comments/1355137064/reactions
2022-12-16T15:59:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1355137064
https://avatars.githubusercontent.com/u/50033854?v=4
https://api.github.com/users/cgpeltier/events{/privacy}
https://api.github.com/users/cgpeltier/followers
https://api.github.com/users/cgpeltier/following{/other_user}
https://api.github.com/users/cgpeltier/gists{/gist_id}
https://github.com/cgpeltier
cgpeltier
MDQ6VXNlcjUwMDMzODU0
https://api.github.com/users/cgpeltier/orgs
https://api.github.com/users/cgpeltier/received_events
https://api.github.com/users/cgpeltier/repos
https://api.github.com/users/cgpeltier/starred{/owner}{/repo}
https://api.github.com/users/cgpeltier/subscriptions
User
https://api.github.com/users/cgpeltier
public
NONE
I am also receiving this error on Sagemaker but not locally, I have noticed that this occurs when the `.dataset/` folder does not contain a single file like:

`dataset.arrow`

but instead contains multiple files like:

`data-00000-of-00002.arrow`
`data-00001-of-00002.arrow`

I think that it may have something to do with this recent PR that updated the behaviour of `dataset.save_to_disk` by introducing sharding: https://github.com/huggingface/datasets/pull/5268

For now I can get around this by forcing datasets==2.8.0 on machine that creates dataset and in the huggingface instance for training (by running this at the start of training script `os.system("pip install datasets==2.8.0")`)

To ensure the dataset is a single shard when saving the dataset locally:

```python3
dataset.flatten_indices().save_to_disk('path/to/dataset', num_shards=1)
```

 and then manually changing the name afterwards from `path/to/dataset/data-00000-of-00001.arrow` to `path/to/dataset/dataset.arrow` and updating the `path/to/dataset/state.json` to reflect this name change. i.e. by changing `state.json` to this:

```javascript
{
  "_data_files": [
    {
      "filename": "dataset.arrow"
    }
  ],
  "_fingerprint": "420086f0636f8727",
  "_format_columns": null,
  "_format_kwargs": {},
  "_format_type": null,
  "_output_all_columns": false,
  "_split": null
}
```
2022-12-22T19:18:37Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1363260948
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5RQbYU
https://api.github.com/repos/huggingface/datasets/issues/comments/1363260948/reactions
2023-01-03T15:26:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1363260948
https://avatars.githubusercontent.com/u/90614572?v=4
https://api.github.com/users/mattdeeperinsights/events{/privacy}
https://api.github.com/users/mattdeeperinsights/followers
https://api.github.com/users/mattdeeperinsights/following{/other_user}
https://api.github.com/users/mattdeeperinsights/gists{/gist_id}
https://github.com/mattdeeperinsights
mattdeeperinsights
MDQ6VXNlcjkwNjE0NTcy
https://api.github.com/users/mattdeeperinsights/orgs
https://api.github.com/users/mattdeeperinsights/received_events
https://api.github.com/users/mattdeeperinsights/repos
https://api.github.com/users/mattdeeperinsights/starred{/owner}{/repo}
https://api.github.com/users/mattdeeperinsights/subscriptions
User
https://api.github.com/users/mattdeeperinsights
public
NONE
Does anyone know if this has been resolved?
2023-12-14T14:20:26Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-1855939546
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps5un2Pa
https://api.github.com/repos/huggingface/datasets/issues/comments/1855939546/reactions
2023-12-14T14:20:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1855939546
https://avatars.githubusercontent.com/u/24214261?v=4
https://api.github.com/users/svanhvitlilja/events{/privacy}
https://api.github.com/users/svanhvitlilja/followers
https://api.github.com/users/svanhvitlilja/following{/other_user}
https://api.github.com/users/svanhvitlilja/gists{/gist_id}
https://github.com/svanhvitlilja
svanhvitlilja
MDQ6VXNlcjI0MjE0MjYx
https://api.github.com/users/svanhvitlilja/orgs
https://api.github.com/users/svanhvitlilja/received_events
https://api.github.com/users/svanhvitlilja/repos
https://api.github.com/users/svanhvitlilja/starred{/owner}{/repo}
https://api.github.com/users/svanhvitlilja/subscriptions
User
https://api.github.com/users/svanhvitlilja
public
NONE
I have the same issue in datasets version 2.3.2
2024-05-12T07:40:02Z
https://github.com/huggingface/datasets/issues/5150#issuecomment-2106152697
https://api.github.com/repos/huggingface/datasets/issues/5150
IC_kwDODunzps59iVb5
https://api.github.com/repos/huggingface/datasets/issues/comments/2106152697/reactions
2024-05-12T07:40:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2106152697
https://avatars.githubusercontent.com/u/66538807?v=4
https://api.github.com/users/EdwardChang5467/events{/privacy}
https://api.github.com/users/EdwardChang5467/followers
https://api.github.com/users/EdwardChang5467/following{/other_user}
https://api.github.com/users/EdwardChang5467/gists{/gist_id}
https://github.com/EdwardChang5467
EdwardChang5467
MDQ6VXNlcjY2NTM4ODA3
https://api.github.com/users/EdwardChang5467/orgs
https://api.github.com/users/EdwardChang5467/received_events
https://api.github.com/users/EdwardChang5467/repos
https://api.github.com/users/EdwardChang5467/starred{/owner}{/repo}
https://api.github.com/users/EdwardChang5467/subscriptions
User
https://api.github.com/users/EdwardChang5467
public
### Describe the bug

Loading a dataset_dict from disk with `load_from_disk` is now creating a `KeyError "length"` that was not occurring in v2.5.2.

Context: 
- Each individual dataset in the dict is created with `Dataset.from_pandas` 
- The dataset_dict is create from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- The pandas dataframe, besides text columns, has a column with a dictionary inside and potentially different keys in each row. Correctly the `Dataset.from_pandas` function adds `key: None` to all dictionaries in each row so that the schema can be correctly inferred.

### Steps to reproduce the bug

Steps to reproduce:

- Upgrade to datasets==2.6.1
- Create a dataset from pandas dataframe with `Dataset.from_pandas`
- Create a dataset_dict from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- Save to disk with the `save` function

### Expected behavior

Same as in v2.5.2, that is load from disk without errors

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.4.209-129.367.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1