ExpectedMoreSplits error when loading C4 dataset
MEMBER
Hi ! We updated the `allenai/c4` repository to allow people to specify which language to load easily (the the [c4 dataset page](https://huggingface.co/datasets/allenai/c4))

To fix this issue **you can update** `datasets` and remove the mention of the legacy configuration name "allenai--c4":

```python
traindata = load_dataset('allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
```
2024-03-21T14:06:06Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2012393426
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps538q_S
https://api.github.com/repos/huggingface/datasets/issues/comments/2012393426/reactions
2024-04-09T16:34:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2012393426
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Did you solve this problem？I have the same bug.It is no use to delete "allenai--c4".
2024-04-01T05:12:24Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2029170578
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps548q-S
https://api.github.com/repos/huggingface/datasets/issues/comments/2029170578/reactions
2024-04-01T05:12:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2029170578
https://avatars.githubusercontent.com/u/95854130?v=4
https://api.github.com/users/K-THU/events{/privacy}
https://api.github.com/users/K-THU/followers
https://api.github.com/users/K-THU/following{/other_user}
https://api.github.com/users/K-THU/gists{/gist_id}
https://github.com/K-THU
K-THU
U_kgDOBbaeMg
https://api.github.com/users/K-THU/orgs
https://api.github.com/users/K-THU/received_events
https://api.github.com/users/K-THU/repos
https://api.github.com/users/K-THU/starred{/owner}{/repo}
https://api.github.com/users/K-THU/subscriptions
User
https://api.github.com/users/K-THU
public
NONE
Did you solve it? I met this problem too.
2024-04-09T07:30:55Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2044327401
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps552fXp
https://api.github.com/repos/huggingface/datasets/issues/comments/2044327401/reactions
2024-04-09T07:30:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2044327401
https://avatars.githubusercontent.com/u/24096174?v=4
https://api.github.com/users/xuChenSJTU/events{/privacy}
https://api.github.com/users/xuChenSJTU/followers
https://api.github.com/users/xuChenSJTU/following{/other_user}
https://api.github.com/users/xuChenSJTU/gists{/gist_id}
https://github.com/xuChenSJTU
xuChenSJTU
MDQ6VXNlcjI0MDk2MTc0
https://api.github.com/users/xuChenSJTU/orgs
https://api.github.com/users/xuChenSJTU/received_events
https://api.github.com/users/xuChenSJTU/repos
https://api.github.com/users/xuChenSJTU/starred{/owner}{/repo}
https://api.github.com/users/xuChenSJTU/subscriptions
User
https://api.github.com/users/xuChenSJTU
public
NONE
But after I romove allenai--c4,it still fails
2024-04-21T02:54:43Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2067870836
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps57QTR0
https://api.github.com/repos/huggingface/datasets/issues/comments/2067870836/reactions
2024-04-21T02:54:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2067870836
https://avatars.githubusercontent.com/u/107231861?v=4
https://api.github.com/users/ssy-small-white/events{/privacy}
https://api.github.com/users/ssy-small-white/followers
https://api.github.com/users/ssy-small-white/following{/other_user}
https://api.github.com/users/ssy-small-white/gists{/gist_id}
https://github.com/ssy-small-white
ssy-small-white
U_kgDOBmQ6dQ
https://api.github.com/users/ssy-small-white/orgs
https://api.github.com/users/ssy-small-white/received_events
https://api.github.com/users/ssy-small-white/repos
https://api.github.com/users/ssy-small-white/starred{/owner}{/repo}
https://api.github.com/users/ssy-small-white/subscriptions
User
https://api.github.com/users/ssy-small-white
public
NONE
For me it works this way. I'm using datasets version 2.17.0
2024-04-22T16:30:14Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2070135775
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps57Y8Pf
https://api.github.com/repos/huggingface/datasets/issues/comments/2070135775/reactions
2024-04-22T16:30:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2070135775
https://avatars.githubusercontent.com/u/105221763?v=4
https://api.github.com/users/davidbhoffmann/events{/privacy}
https://api.github.com/users/davidbhoffmann/followers
https://api.github.com/users/davidbhoffmann/following{/other_user}
https://api.github.com/users/davidbhoffmann/gists{/gist_id}
https://github.com/davidbhoffmann
davidbhoffmann
U_kgDOBkWOgw
https://api.github.com/users/davidbhoffmann/orgs
https://api.github.com/users/davidbhoffmann/received_events
https://api.github.com/users/davidbhoffmann/repos
https://api.github.com/users/davidbhoffmann/starred{/owner}{/repo}
https://api.github.com/users/davidbhoffmann/subscriptions
User
https://api.github.com/users/davidbhoffmann
public
NONE
First,  pip install --upgrade datasets.
Second, Update the following two lines of code in data.py (in lib)
traindata = load_dataset('allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
2024-07-25T16:33:41Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2250905109
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps6GKhYV
https://api.github.com/repos/huggingface/datasets/issues/comments/2250905109/reactions
2024-07-25T16:33:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2250905109
https://avatars.githubusercontent.com/u/55442242?v=4
https://api.github.com/users/chaosright/events{/privacy}
https://api.github.com/users/chaosright/followers
https://api.github.com/users/chaosright/following{/other_user}
https://api.github.com/users/chaosright/gists{/gist_id}
https://github.com/chaosright
chaosright
MDQ6VXNlcjU1NDQyMjQy
https://api.github.com/users/chaosright/orgs
https://api.github.com/users/chaosright/received_events
https://api.github.com/users/chaosright/repos
https://api.github.com/users/chaosright/starred{/owner}{/repo}
https://api.github.com/users/chaosright/subscriptions
User
https://api.github.com/users/chaosright
public
MEMBER
The error is in the Wanda repository: https://github.com/locuslab/wanda
- https://github.com/locuslab/wanda/issues/57

Concretely, in these code lines:
https://github.com/locuslab/wanda/blob/8e8fc87b4a2f9955baa7e76e64d5fce7fa8724a6/lib/data.py#L43-L44

Please report there and/or make the fix in their code.
2024-07-29T07:21:08Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2255142335
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps6Gar2_
https://api.github.com/repos/huggingface/datasets/issues/comments/2255142335/reactions
2024-07-29T07:21:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2255142335
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
> traindata = load_dataset('allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
> valdata = load_dataset('allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')

Solved for me ! Thanks!
2024-09-18T19:57:12Z
https://github.com/huggingface/datasets/issues/6746#issuecomment-2359294581
https://api.github.com/repos/huggingface/datasets/issues/6746
IC_kwDODunzps6Mn_p1
https://api.github.com/repos/huggingface/datasets/issues/comments/2359294581/reactions
2024-09-18T19:57:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2359294581
https://avatars.githubusercontent.com/u/180708858?v=4
https://api.github.com/users/SimWangArizona/events{/privacy}
https://api.github.com/users/SimWangArizona/followers
https://api.github.com/users/SimWangArizona/following{/other_user}
https://api.github.com/users/SimWangArizona/gists{/gist_id}
https://github.com/SimWangArizona
SimWangArizona
U_kgDOCsVl-g
https://api.github.com/users/SimWangArizona/orgs
https://api.github.com/users/SimWangArizona/received_events
https://api.github.com/users/SimWangArizona/repos
https://api.github.com/users/SimWangArizona/starred{/owner}{/repo}
https://api.github.com/users/SimWangArizona/subscriptions
User
https://api.github.com/users/SimWangArizona
public
### Describe the bug

I encounter bug when running the example command line 

```python
    python main.py \
    --model decapoda-research/llama-7b-hf \
    --prune_method wanda \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --save out/llama_7b/unstructured/wanda/ 
```

The bug occurred at these lines of code (when loading c4 dataset) 

```python
traindata = load_dataset('allenai/c4', 'allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', 'allenai--c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
```

The error message states:
```    
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))                                                                           
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```



### Steps to reproduce the bug

1. I encounter bug when running the example command line 


### Expected behavior

The error message states:
```    
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))                                                                           
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```



### Environment info

I'm using cuda 12.4, so I use ```pip install pytorch``` instead of conda provided in install.md

Also, I've tried another environment using the same commands in install.md, but the same bug occured 