500 internal server error when trying to open a dataset composed of Zarr stores
MEMBER
Hi @jacobbieker, thanks for reporting!

I have transferred this issue to our Hub team and they are investigating it. I keep you informed. 
2022-03-04T15:21:52Z
https://github.com/huggingface/datasets/issues/3823#issuecomment-1059257318
https://api.github.com/repos/huggingface/datasets/issues/3823
IC_kwDODunzps4_Ivvm
https://api.github.com/repos/huggingface/datasets/issues/comments/1059257318/reactions
2022-03-04T15:21:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1059257318
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Hi @jacobbieker, we are investigating this issue on our side and we'll see if we can fix it, but please note that your repo is considered problematic for git. Here are the results of running https://github.com/github/git-sizer on it:

```
Processing blobs: 147448                        
Processing trees: 27                        
Processing commits: 4                        
Matching commits to trees: 4                        
Processing annotated tags: 0                        
Processing references: 3                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |   167 k   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of files        [2] |   189 k   | ***                            |

[1]  aa057d2667c34c70c6146efc631f5c9917ff326e (refs/heads/main:2016.zarr/unknown)
[2]  6897b7bf6440fdd16b2c39d08085a669e7eaa59d (refs/heads/main^{tree})
```

You can check https://github.com/github/git-sizer for more information on how to avoid such pathological structures.
2022-03-04T15:32:24Z
https://github.com/huggingface/datasets/issues/3823#issuecomment-1059265998
https://api.github.com/repos/huggingface/datasets/issues/3823
IC_kwDODunzps4_Ix3O
https://api.github.com/repos/huggingface/datasets/issues/comments/1059265998/reactions
2022-03-04T15:32:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1059265998
https://avatars.githubusercontent.com/u/5020707?v=4
https://api.github.com/users/Pierrci/events{/privacy}
https://api.github.com/users/Pierrci/followers
https://api.github.com/users/Pierrci/following{/other_user}
https://api.github.com/users/Pierrci/gists{/gist_id}
https://github.com/Pierrci
Pierrci
MDQ6VXNlcjUwMjA3MDc=
https://api.github.com/users/Pierrci/orgs
https://api.github.com/users/Pierrci/received_events
https://api.github.com/users/Pierrci/repos
https://api.github.com/users/Pierrci/starred{/owner}{/repo}
https://api.github.com/users/Pierrci/subscriptions
User
https://api.github.com/users/Pierrci
public
NONE
Hi, thanks for getting back to me so quick! And yeah, I figured that was probably the problem. I was going to try to delete the repo, but couldn't through the website, so if that's the easiest way to solve it, I can regenerate the dataset in a different format with less tiny files, and you guys can delete the repo as it is. Zarr just saves everything as lots of small files to make chunks easy to load, which is why I was preferring that format, but maybne that just doesn't work well for HF datasets.
2022-03-04T15:38:48Z
https://github.com/huggingface/datasets/issues/3823#issuecomment-1059271133
https://api.github.com/repos/huggingface/datasets/issues/3823
IC_kwDODunzps4_IzHd
https://api.github.com/repos/huggingface/datasets/issues/comments/1059271133/reactions
2022-03-04T15:38:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1059271133
https://avatars.githubusercontent.com/u/7170359?v=4
https://api.github.com/users/jacobbieker/events{/privacy}
https://api.github.com/users/jacobbieker/followers
https://api.github.com/users/jacobbieker/following{/other_user}
https://api.github.com/users/jacobbieker/gists{/gist_id}
https://github.com/jacobbieker
jacobbieker
MDQ6VXNlcjcxNzAzNTk=
https://api.github.com/users/jacobbieker/orgs
https://api.github.com/users/jacobbieker/received_events
https://api.github.com/users/jacobbieker/repos
https://api.github.com/users/jacobbieker/starred{/owner}{/repo}
https://api.github.com/users/jacobbieker/subscriptions
User
https://api.github.com/users/jacobbieker
public
MEMBER
Hi @jacobbieker,

For future use cases, our Hub team is still pondering whether to limit the maximum number of files per repo to avoid technical issues...

On the meantime, they have made a fix and your dataset is working: https://huggingface.co/datasets/openclimatefix/mrms
2022-03-08T09:47:39Z
https://github.com/huggingface/datasets/issues/3823#issuecomment-1061592094
https://api.github.com/repos/huggingface/datasets/issues/3823
IC_kwDODunzps4_Rpwe
https://api.github.com/repos/huggingface/datasets/issues/comments/1061592094/reactions
2022-03-08T09:47:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061592094
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine. 

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format? 

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0

