Loading huggan/CelebA-HQ throws pyarrow.lib.ArrowInvalid
COLLABORATOR
Hi! IIRC one of the files in this dataset is corrupted due to https://github.com/huggingface/datasets/pull/4081 (fixed now).

@NielsRogge Could you please re-generate and re-push this dataset (or I can do it if you share the generation script)?
2022-09-08T16:29:04Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1240947119
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5J91mv
https://api.github.com/repos/huggingface/datasets/issues/comments/1240947119/reactions
2022-09-08T16:29:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1240947119
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Could you put something in place to catch these problems?  I'm seeing this on another dataset consistently too and I guess I can't fix it in code?
2023-01-17T11:36:17Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1385293042
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5SkeTy
https://api.github.com/repos/huggingface/datasets/issues/comments/1385293042/reactions
2023-01-17T11:36:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1385293042
https://avatars.githubusercontent.com/u/445208?v=4
https://api.github.com/users/alexjc/events{/privacy}
https://api.github.com/users/alexjc/followers
https://api.github.com/users/alexjc/following{/other_user}
https://api.github.com/users/alexjc/gists{/gist_id}
https://github.com/alexjc
alexjc
MDQ6VXNlcjQ0NTIwOA==
https://api.github.com/users/alexjc/orgs
https://api.github.com/users/alexjc/received_events
https://api.github.com/users/alexjc/repos
https://api.github.com/users/alexjc/starred{/owner}{/repo}
https://api.github.com/users/alexjc/subscriptions
User
https://api.github.com/users/alexjc
public
CONTRIBUTOR
Hey,

Yes the notebook I used to upload this dataset can be found here: https://colab.research.google.com/drive/141LJCcM2XyqprPY83nIQ-Zk3BbxWeahq?usp=sharing.

If you have time to regenerate the dataset, would be great.
2023-01-17T13:00:42Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1385389391
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5Sk11P
https://api.github.com/repos/huggingface/datasets/issues/comments/1385389391/reactions
2023-01-17T13:07:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1385389391
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
Sorry, maybe I wasn't clear enough that it's a different dataset `laion2B-multi-joined-translated-to-en`.  I think there should be checks in the upload, tests on the server, or validation after download (hashes) to catch these problems.

Lots of bandwidth wasted otherwise! /cc @mariosasko
2023-01-17T15:13:31Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1385579817
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5SlkUp
https://api.github.com/repos/huggingface/datasets/issues/comments/1385579817/reactions
2023-01-17T15:13:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1385579817
https://avatars.githubusercontent.com/u/445208?v=4
https://api.github.com/users/alexjc/events{/privacy}
https://api.github.com/users/alexjc/followers
https://api.github.com/users/alexjc/following{/other_user}
https://api.github.com/users/alexjc/gists{/gist_id}
https://github.com/alexjc
alexjc
MDQ6VXNlcjQ0NTIwOA==
https://api.github.com/users/alexjc/orgs
https://api.github.com/users/alexjc/received_events
https://api.github.com/users/alexjc/repos
https://api.github.com/users/alexjc/starred{/owner}{/repo}
https://api.github.com/users/alexjc/subscriptions
User
https://api.github.com/users/alexjc
public
CONTRIBUTOR
Yes @alexjc sorry was more a reply to @JeanKaddour.

And indeed it'd be great to have additional checks to avoid these errors. 
2023-01-17T15:23:29Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1385594697
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5Sln9J
https://api.github.com/repos/huggingface/datasets/issues/comments/1385594697/reactions
2023-01-17T15:23:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1385594697
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
COLLABORATOR
cc @severo since such checks should probably be implemented on the datasets-server side.
2023-01-17T16:34:53Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1385700166
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5SmBtG
https://api.github.com/repos/huggingface/datasets/issues/comments/1385700166/reactions
2023-01-17T16:34:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1385700166
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi,

It seems the problem is still persist. I have encountered the exact same problem using just 2 line of code above. 

The error code is as follows:

```
發生例外狀況: DatasetGenerationError
An error occurred while generating the dataset
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

  File "/code/ddpm_learn/train.py", line 65, in <module>
    dataset = load_dataset("huggan/CelebA-HQ", cache_dir="./CelebA-HQ"
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```
2023-02-01T11:08:19Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1411880516
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5UJ5ZE
https://api.github.com/repos/huggingface/datasets/issues/comments/1411880516/reactions
2023-02-01T11:08:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1411880516
https://avatars.githubusercontent.com/u/38370649?v=4
https://api.github.com/users/allanchan339/events{/privacy}
https://api.github.com/users/allanchan339/followers
https://api.github.com/users/allanchan339/following{/other_user}
https://api.github.com/users/allanchan339/gists{/gist_id}
https://github.com/allanchan339
allanchan339
MDQ6VXNlcjM4MzcwNjQ5
https://api.github.com/users/allanchan339/orgs
https://api.github.com/users/allanchan339/received_events
https://api.github.com/users/allanchan339/repos
https://api.github.com/users/allanchan339/starred{/owner}{/repo}
https://api.github.com/users/allanchan339/subscriptions
User
https://api.github.com/users/allanchan339
public
CONTRIBUTOR
Yes for the moment refer to the notebook linked above if you want to create a HF dataset yourself
2023-02-01T11:14:10Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1411888799
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5UJ7af
https://api.github.com/repos/huggingface/datasets/issues/comments/1411888799/reactions
2023-02-01T11:14:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1411888799
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
Hi @NielsRogge ,
I can help to push the dataset to the cloud. However, I cannot locate the situation so far. I wonder if 
1. the downloaded files so far has corruption s.t. the file cannot generate properly, or
2. the downloaded files has no bug, the bug is caused by buggy upload program so that I can use what I have just downloaded to re-upload to cloud

Thank, 
Allan
2023-02-02T02:40:53Z
https://github.com/huggingface/datasets/issues/4886#issuecomment-1413070300
https://api.github.com/repos/huggingface/datasets/issues/4886
IC_kwDODunzps5UOb3c
https://api.github.com/repos/huggingface/datasets/issues/comments/1413070300/reactions
2023-02-02T02:40:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413070300
https://avatars.githubusercontent.com/u/38370649?v=4
https://api.github.com/users/allanchan339/events{/privacy}
https://api.github.com/users/allanchan339/followers
https://api.github.com/users/allanchan339/following{/other_user}
https://api.github.com/users/allanchan339/gists{/gist_id}
https://github.com/allanchan339
allanchan339
MDQ6VXNlcjM4MzcwNjQ5
https://api.github.com/users/allanchan339/orgs
https://api.github.com/users/allanchan339/received_events
https://api.github.com/users/allanchan339/repos
https://api.github.com/users/allanchan339/starred{/owner}{/repo}
https://api.github.com/users/allanchan339/subscriptions
User
https://api.github.com/users/allanchan339
public
## Describe the bug
Loading huggan/CelebA-HQ throws pyarrow.lib.ArrowInvalid

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('huggan/CelebA-HQ')
```

## Expected results
See https://colab.research.google.com/drive/141LJCcM2XyqprPY83nIQ-Zk3BbxWeahq?usp=sharing#scrollTo=N3ml_7f8kzDd

## Actual results
```
  File "/home/jean/projects/cold_diffusion/celebA.py", line 4, in <module>
    dataset = load_dataset('huggan/CelebA-HQ')
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/load.py", line 1793, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/builder.py", line 1274, in _prepare_split
    for key, table in logging.tqdm(
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 67, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/pyarrow/parquet/__init__.py", line 286, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1227, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.4.1.dev0
- Platform: Ubuntu 18.04
- Python version: 3.10
- PyArrow version: pyarrow 9.0.0

