Datasets crashing runs due to KeyError
NONE
i once had the same error and I could fix that by pushing a fake or a dummy commit on my hugging face dataset repo
2023-08-15T18:34:08Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1679412054
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5kGctW
https://api.github.com/repos/huggingface/datasets/issues/comments/1679412054/reactions
2023-08-15T18:34:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1679412054
https://avatars.githubusercontent.com/u/59269023?v=4
https://api.github.com/users/erfanzar/events{/privacy}
https://api.github.com/users/erfanzar/followers
https://api.github.com/users/erfanzar/following{/other_user}
https://api.github.com/users/erfanzar/gists{/gist_id}
https://github.com/erfanzar
erfanzar
MDQ6VXNlcjU5MjY5MDIz
https://api.github.com/users/erfanzar/orgs
https://api.github.com/users/erfanzar/received_events
https://api.github.com/users/erfanzar/repos
https://api.github.com/users/erfanzar/starred{/owner}{/repo}
https://api.github.com/users/erfanzar/subscriptions
User
https://api.github.com/users/erfanzar
public
COLLABORATOR
Hi! We need a reproducer to fix this. Can you provide a link to the dataset (if it's public)?
2023-08-17T16:32:57Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1682605522
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5kSoXS
https://api.github.com/repos/huggingface/datasets/issues/comments/1682605522/reactions
2023-08-17T16:32:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682605522
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> Hi! We need a reproducer to fix this. Can you provide a link to the dataset (if it's public)?

Hi Mario,

Unfortunately, the dataset in question is currently private until the model is trained and released.

This is not happening with one dataset but numerous hosted private datasets.

I am only loading the dataset and doing nothing else currently. It seems to happen completely sporadically.

Thank you,

Enrico
2023-08-20T17:33:03Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1685348598
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5kdGD2
https://api.github.com/repos/huggingface/datasets/issues/comments/1685348598/reactions
2023-08-20T17:33:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1685348598
https://avatars.githubusercontent.com/u/25208228?v=4
https://api.github.com/users/conceptofmind/events{/privacy}
https://api.github.com/users/conceptofmind/followers
https://api.github.com/users/conceptofmind/following{/other_user}
https://api.github.com/users/conceptofmind/gists{/gist_id}
https://github.com/conceptofmind
conceptofmind
MDQ6VXNlcjI1MjA4MjI4
https://api.github.com/users/conceptofmind/orgs
https://api.github.com/users/conceptofmind/received_events
https://api.github.com/users/conceptofmind/repos
https://api.github.com/users/conceptofmind/starred{/owner}{/repo}
https://api.github.com/users/conceptofmind/subscriptions
User
https://api.github.com/users/conceptofmind
public
NONE
Hi,

I have the same error in the dataset viewer with my dataset
https://huggingface.co/datasets/elsaEU/ELSA10M_track1

Has anyone solved this issue?

Edit: After a dummy commit the error changed in ConfigNamesError
2023-10-12T09:03:37Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1759218724
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5o24wk
https://api.github.com/repos/huggingface/datasets/issues/comments/1759218724/reactions
2023-10-12T09:12:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1759218724
https://avatars.githubusercontent.com/u/9029817?v=4
https://api.github.com/users/rs9000/events{/privacy}
https://api.github.com/users/rs9000/followers
https://api.github.com/users/rs9000/following{/other_user}
https://api.github.com/users/rs9000/gists{/gist_id}
https://github.com/rs9000
rs9000
MDQ6VXNlcjkwMjk4MTc=
https://api.github.com/users/rs9000/orgs
https://api.github.com/users/rs9000/received_events
https://api.github.com/users/rs9000/repos
https://api.github.com/users/rs9000/starred{/owner}{/repo}
https://api.github.com/users/rs9000/subscriptions
User
https://api.github.com/users/rs9000
public
COLLABORATOR
@rs9000 The problem seems to be the (large) number of commits, as explained in https://huggingface.co/docs/hub/repositories-recommendations. This can be fixed by running:
```python
import huggingface_hub
huggingface_hub.super_squash_history(repo_id="elsaEU/ELSA10M_track1")
``` 

The issue stems from `push_to_hub` creating one commit per shard -  https://github.com/huggingface/datasets/pull/6269 should fix this issue (will create one commit per 50 uploaded shards by default). The linked PR will be included in the next `datasets` release.


cc @lhoestq @severo for visibility
2023-10-12T14:03:10Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1759674352
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5o4n_w
https://api.github.com/repos/huggingface/datasets/issues/comments/1759674352/reactions
2023-10-12T14:03:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1759674352
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thank you @mariosasko it works.
2023-10-12T14:44:28Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1759754546
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5o47ky
https://api.github.com/repos/huggingface/datasets/issues/comments/1759754546/reactions
2023-10-12T14:44:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1759754546
https://avatars.githubusercontent.com/u/9029817?v=4
https://api.github.com/users/rs9000/events{/privacy}
https://api.github.com/users/rs9000/followers
https://api.github.com/users/rs9000/following{/other_user}
https://api.github.com/users/rs9000/gists{/gist_id}
https://github.com/rs9000
rs9000
MDQ6VXNlcjkwMjk4MTc=
https://api.github.com/users/rs9000/orgs
https://api.github.com/users/rs9000/received_events
https://api.github.com/users/rs9000/repos
https://api.github.com/users/rs9000/starred{/owner}{/repo}
https://api.github.com/users/rs9000/subscriptions
User
https://api.github.com/users/rs9000
public
COLLABORATOR
#6269 has been merged, so I'm closing this issue
2023-11-30T16:28:57Z
https://github.com/huggingface/datasets/issues/6124#issuecomment-1834112318
https://api.github.com/repos/huggingface/datasets/issues/6124
IC_kwDODunzps5tUlU-
https://api.github.com/repos/huggingface/datasets/issues/comments/1834112318/reactions
2023-11-30T16:28:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1834112318
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Describe the bug

Hi all,

I have been running into a pretty persistent issue recently when trying to load datasets. 
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```
I receive a KeyError which crashes the runs.
```
Traceback (most recent call last):
    main()

    train_dataset = load_dataset(
                    ^^^^^^^^^^^^^
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
    raise e1 from None

    ).get_module()
      ^^^^^^^^^^^^
    else get_data_patterns(base_path, download_config=self.download_config)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return _get_data_files_patterns(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    data_files = pattern_resolver(pattern)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    fs, _, _ = get_fs_token_paths(pattern, storage_options=storage_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    paths = [f for f in sorted(fs.glob(paths)) if not fs.isdir(f)]
                               ^^^^^^^^^^^^^^

    allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    for _, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):

    listing = self.ls(path, detail=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    "last_modified": parse_datetime(tree_item["lastCommit"]["date"]),
                                    ~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'lastCommit'
```
Any help would be greatly appreciated.

Thank you,

Enrico

### Steps to reproduce the bug

Load the dataset from the Huggingface hub.
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```

### Expected behavior

Loads the dataset.

### Environment info

datasets-2.14.3
CUDA 11.8
Python 3.11