Support GeoParquet
NONE
Thank you, @severo ! I would be more than happy to help in any way I can. I am not familiar with this repo's codebase, but I would be eager to contribute. :)

For the preview in Datasets Hub, I think it makes sense to just display the geospatial column as text. If there were a dataset loader, though, I think it should be able to support the geospatial components. Geopandas is probably the most user-friendly interface for that. I'm not sure if it's currently relevant in the context of geoparquet, but I think the pyogrio driver is faster than fiona.

But the whole gdal dependency thing can be a real pain. If anything, it would need to be an optional dependency. Maybe it would be best if the loader tries importing relevant geospatial libraries, and in the event of an ImportError, falls back to text for the geometry column.

Please let me know if I can be of assistance, and thanks again for creating this Issue. :)
2023-11-20T14:10:23Z
https://github.com/huggingface/datasets/issues/6438#issuecomment-1819133593
https://api.github.com/repos/huggingface/datasets/issues/6438
IC_kwDODunzps5sbcaZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1819133593/reactions
2023-11-20T14:10:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1819133593
https://avatars.githubusercontent.com/u/84336755?v=4
https://api.github.com/users/joshuasundance-swca/events{/privacy}
https://api.github.com/users/joshuasundance-swca/followers
https://api.github.com/users/joshuasundance-swca/following{/other_user}
https://api.github.com/users/joshuasundance-swca/gists{/gist_id}
https://github.com/joshuasundance-swca
joshuasundance-swca
MDQ6VXNlcjg0MzM2NzU1
https://api.github.com/users/joshuasundance-swca/orgs
https://api.github.com/users/joshuasundance-swca/received_events
https://api.github.com/users/joshuasundance-swca/repos
https://api.github.com/users/joshuasundance-swca/starred{/owner}{/repo}
https://api.github.com/users/joshuasundance-swca/subscriptions
User
https://api.github.com/users/joshuasundance-swca
public
CONTRIBUTOR
Just hitting into this same issue too showing GeoParquet files in Datasets Viewer. I tried to implement a custom reader for GeoParquet in https://huggingface.co/datasets/weiji14/clay_vector_embeddings/discussions/1, but it seems like HuggingFace has disabled datasets with custom loading scripts from using the dataset viewer according to https://discuss.huggingface.co/t/dataset-repo-requires-arbitrary-python-code-execution/59346 :frowning_face: 

![image](https://github.com/huggingface/datasets/assets/23487320/2f84d8ce-91c2-48cb-b72c-547ea8583892)

I'm thinking now if there's a way to simply map files with GeoParquet extensions (*.gpq, *.geoparquet, etc) to use the Parquet reader. Maybe we could allowlist these geoparquet file extensions at https://github.com/huggingface/datasets/blame/0caf91285116ec910f409e82cc6e1f4cff7496e3/src/datasets/packaged_modules/__init__.py#L30-L51? Having the table columns show up would be a quick win.

Longer term though, it would certainly be nice if the WKB geometry columns could be displayed in a nicer form. Geopandas' [read_parquet](https://geopandas.org/en/v0.14.1/docs/reference/api/geopandas.read_parquet.html) function is supposedly faster than `pyogrio.read_dataframe` according to https://github.com/geopandas/geopandas/discussions/2724#discussioncomment-4606048, but there's also [`pyogrio.raw.read_arrow`](https://pyogrio.readthedocs.io/en/latest/api.html#pyogrio.raw.read_arrow) now that can read into a `pyarrow.Table` directly.
2023-12-18T03:57:30Z
https://github.com/huggingface/datasets/issues/6438#issuecomment-1859511902
https://api.github.com/repos/huggingface/datasets/issues/6438
IC_kwDODunzps5u1eZe
https://api.github.com/repos/huggingface/datasets/issues/comments/1859511902/reactions
2023-12-18T07:33:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1859511902
https://avatars.githubusercontent.com/u/23487320?v=4
https://api.github.com/users/weiji14/events{/privacy}
https://api.github.com/users/weiji14/followers
https://api.github.com/users/weiji14/following{/other_user}
https://api.github.com/users/weiji14/gists{/gist_id}
https://github.com/weiji14
weiji14
MDQ6VXNlcjIzNDg3MzIw
https://api.github.com/users/weiji14/orgs
https://api.github.com/users/weiji14/received_events
https://api.github.com/users/weiji14/repos
https://api.github.com/users/weiji14/starred{/owner}{/repo}
https://api.github.com/users/weiji14/subscriptions
User
https://api.github.com/users/weiji14
public
CONTRIBUTOR
Update: It looks like renaming the GeoParquet file to have a file extension of `*.parquet` works (see https://huggingface.co/datasets/weiji14/clay_vector_embeddings). HuggingFace's default parquet reader is able to read the GeoParquet file, though the geometry column is of an unknown type:

![image](https://github.com/huggingface/datasets/assets/23487320/9060c300-d595-4409-9ccb-5e0207396883)

I've opened a quick PR at #6508 to allow files with a `*.geoparquet` or `*.gpq` extension to be read using the default Parquet reader. Let's see how that goes :smile:
2023-12-18T05:06:22Z
https://github.com/huggingface/datasets/issues/6438#issuecomment-1859566294
https://api.github.com/repos/huggingface/datasets/issues/6438
IC_kwDODunzps5u1rrW
https://api.github.com/repos/huggingface/datasets/issues/comments/1859566294/reactions
2024-02-07T02:47:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1859566294
https://avatars.githubusercontent.com/u/23487320?v=4
https://api.github.com/users/weiji14/events{/privacy}
https://api.github.com/users/weiji14/followers
https://api.github.com/users/weiji14/following{/other_user}
https://api.github.com/users/weiji14/gists{/gist_id}
https://github.com/weiji14
weiji14
MDQ6VXNlcjIzNDg3MzIw
https://api.github.com/users/weiji14/orgs
https://api.github.com/users/weiji14/received_events
https://api.github.com/users/weiji14/repos
https://api.github.com/users/weiji14/starred{/owner}{/repo}
https://api.github.com/users/weiji14/subscriptions
User
https://api.github.com/users/weiji14
public
NONE
@joshuasundance-swca, @weiji14, If I'm understanding this correctly, the code below wouldn't be recommended to due to dependency headaches? If that's the case, what solution would there be to see the geometry features for .gpq files in huggingfaceHub? 

code for dataset_loader.py
```
import geopandas as gpd
# ... (other imports remain the same)

class ClayVectorEmbeddings(datasets.ArrowBasedBuilder):
    # ... (other parts of the class remain the same)

    def _info(self):
        # Read the GeoParquet file to get the schema for the 'geometry' feature
        gdf = gpd.read_file("path/to/your/geoparquet/file.gpq")  # Replace with your file path
        geometry_schema = str(gdf.geometry.dtype)

        return datasets.DatasetInfo(
            # This is the description that will appear on the datasets page.
            description="Clay Vector Embeddings in GeoParquet format.",
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "source_url": datasets.Value(dtype="string"),
                    "date": datasets.Value(dtype="date32"),
                    "embeddings": datasets.Value("string"),
                    "geometry": datasets.Value(dtype=geometry_schema),  # Use the schema read by GeoPandas
                    # ... (other features)
                }
            ),
        )

# ... (rest of the script remains the same)

```
2023-12-30T03:38:11Z
https://github.com/huggingface/datasets/issues/6438#issuecomment-1872441582
https://api.github.com/repos/huggingface/datasets/issues/6438
IC_kwDODunzps5vmzDu
https://api.github.com/repos/huggingface/datasets/issues/comments/1872441582/reactions
2023-12-30T03:38:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1872441582
https://avatars.githubusercontent.com/u/124646200?v=4
https://api.github.com/users/murdadesmaeeli/events{/privacy}
https://api.github.com/users/murdadesmaeeli/followers
https://api.github.com/users/murdadesmaeeli/following{/other_user}
https://api.github.com/users/murdadesmaeeli/gists{/gist_id}
https://github.com/murdadesmaeeli
murdadesmaeeli
U_kgDOB23zOA
https://api.github.com/users/murdadesmaeeli/orgs
https://api.github.com/users/murdadesmaeeli/received_events
https://api.github.com/users/murdadesmaeeli/repos
https://api.github.com/users/murdadesmaeeli/starred{/owner}{/repo}
https://api.github.com/users/murdadesmaeeli/subscriptions
User
https://api.github.com/users/murdadesmaeeli
public
CONTRIBUTOR
Hi @mehrdad-es, I'm not sure if HuggingFace would be keen to add `geopandas` to HuggingFace Hub (maybe a question for @severo?). Having a geometry viewer would be an even bigger task, and if you're thinking of a map-viewer, it might involve some redesign of the website UI. Some of my colleagues are working on streamlining GeoParquet visualization from cloud-hosted instances like HuggingFace (see e.g. https://github.com/developmentseed/lonboard/issues/314), and we could definitely come up with something if there's interest.
2024-02-07T02:48:25Z
https://github.com/huggingface/datasets/issues/6438#issuecomment-1931172559
https://api.github.com/repos/huggingface/datasets/issues/6438
IC_kwDODunzps5zG1rP
https://api.github.com/repos/huggingface/datasets/issues/comments/1931172559/reactions
2024-02-07T02:48:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1931172559
https://avatars.githubusercontent.com/u/23487320?v=4
https://api.github.com/users/weiji14/events{/privacy}
https://api.github.com/users/weiji14/followers
https://api.github.com/users/weiji14/following{/other_user}
https://api.github.com/users/weiji14/gists{/gist_id}
https://github.com/weiji14
weiji14
MDQ6VXNlcjIzNDg3MzIw
https://api.github.com/users/weiji14/orgs
https://api.github.com/users/weiji14/received_events
https://api.github.com/users/weiji14/repos
https://api.github.com/users/weiji14/starred{/owner}{/repo}
https://api.github.com/users/weiji14/subscriptions
User
https://api.github.com/users/weiji14
public
COLLABORATOR
I've created https://github.com/huggingface/datasets-server/issues/2416 to discuss the possibility of supporting (vectorial) geospatial columns in the dataset viewer, or in the converted parquet files.

At the same time, it would be super interesting to see what is already possible to do with a Hugging Face dataset that hosts geospatial data. 

> Some of my colleagues are working on streamlining GeoParquet visualization from cloud-hosted instances like HuggingFace (see e.g. https://github.com/developmentseed/lonboard/issues/314), and we could definitely come up with something if there's interest.

It would be awesome to show this inside a [Space](https://huggingface.co/docs/hub/spaces).
2024-02-07T08:36:50Z
https://github.com/huggingface/datasets/issues/6438#issuecomment-1931531341
https://api.github.com/repos/huggingface/datasets/issues/6438
IC_kwDODunzps5zINRN
https://api.github.com/repos/huggingface/datasets/issues/comments/1931531341/reactions
2024-02-07T08:36:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1931531341
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
### Feature request

Support the GeoParquet format

### Motivation

GeoParquet (https://geoparquet.org/) is a common format for sharing vectorial geospatial data on the cloud, along with "traditional" data columns.

It would be nice to be able to load this format with datasets, and more generally, in the Datasets Hub (see https://huggingface.co/datasets/joshuasundance/govgis_nov2023-slim-spatial/discussions/1).

### Your contribution

I would be happy to help work on a PR (but I don't think I can do one on my own).

Also, we have to define what we want to support:
- load all the columns, but get the "geospatial" column in text-only mode for now
- or, fully support the spatial features, maybe taking inspiration from (or depending upon) https://geopandas.org/en/stable/index.html (which itself depends on https://fiona.readthedocs.io/en/stable/, which requires a local install of https://gdal.org/)