Reload features from Parquet metadata
CONTRIBUTOR
Thanks a lot for your help @lhoestq. I've simplified what turned out to be a simple fix and added the unit test.

Does this look ready to be merged or is there anything I'm still missing?
2023-02-09T16:17:33Z
https://github.com/huggingface/datasets/pull/5516#issuecomment-1424454748
https://api.github.com/repos/huggingface/datasets/issues/5516
IC_kwDODunzps5U53Rc
https://api.github.com/repos/huggingface/datasets/issues/comments/1424454748/reactions
2023-02-09T16:19:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1424454748
https://avatars.githubusercontent.com/u/6368040?v=4
https://api.github.com/users/MFreidank/events{/privacy}
https://api.github.com/users/MFreidank/followers
https://api.github.com/users/MFreidank/following{/other_user}
https://api.github.com/users/MFreidank/gists{/gist_id}
https://github.com/MFreidank
MFreidank
MDQ6VXNlcjYzNjgwNDA=
https://api.github.com/users/MFreidank/orgs
https://api.github.com/users/MFreidank/received_events
https://api.github.com/users/MFreidank/repos
https://api.github.com/users/MFreidank/starred{/owner}{/repo}
https://api.github.com/users/MFreidank/subscriptions
User
https://api.github.com/users/MFreidank
public
MEMBER
Cool ! I think you just need to remove the unused import in `io/parquet.py`
```
src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
```
and we're good to merge :)
2023-02-09T16:26:33Z
https://github.com/huggingface/datasets/pull/5516#issuecomment-1424467173
https://api.github.com/repos/huggingface/datasets/issues/5516
IC_kwDODunzps5U56Tl
https://api.github.com/repos/huggingface/datasets/issues/comments/1424467173/reactions
2023-02-09T16:26:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1424467173
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-02-09T16:28:46Z
https://github.com/huggingface/datasets/pull/5516#issuecomment-1424470413
https://api.github.com/repos/huggingface/datasets/issues/5516
IC_kwDODunzps5U57GN
https://api.github.com/repos/huggingface/datasets/issues/comments/1424470413/reactions
2023-02-12T16:00:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1424470413
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
> Cool ! I think you just need to remove the unused import in `io/parquet.py`
> 
> ```
> src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
> ```
> 
> and we're good to merge :)

Done! Thanks a lot, this was fun :)
2023-02-09T18:12:13Z
https://github.com/huggingface/datasets/pull/5516#issuecomment-1424617902
https://api.github.com/repos/huggingface/datasets/issues/5516
IC_kwDODunzps5U6fGu
https://api.github.com/repos/huggingface/datasets/issues/comments/1424617902/reactions
2023-02-09T18:12:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1424617902
https://avatars.githubusercontent.com/u/6368040?v=4
https://api.github.com/users/MFreidank/events{/privacy}
https://api.github.com/users/MFreidank/followers
https://api.github.com/users/MFreidank/following{/other_user}
https://api.github.com/users/MFreidank/gists{/gist_id}
https://github.com/MFreidank
MFreidank
MDQ6VXNlcjYzNjgwNDA=
https://api.github.com/users/MFreidank/orgs
https://api.github.com/users/MFreidank/received_events
https://api.github.com/users/MFreidank/repos
https://api.github.com/users/MFreidank/starred{/owner}{/repo}
https://api.github.com/users/MFreidank/subscriptions
User
https://api.github.com/users/MFreidank
public
Resolves #5482. 

Attaches feature metadata to parquet files serialised using `Dataset.to_parquet`. 

This allows retrieving data with "rich" feature types (e.g., `datasets.features.image.Image` or `datasets.features.audio.Audio`) from parquet files without cumbersome casting (for an example, see #5482).

@lhoestq It seems that it is sufficient to attach metadata to the schema prior to serialising and features are loaded back with correct types afterwards automatically. 

I used the following script to test the implementation:

```python
from pathlib import Path

import datasets

dataset_name = "Maysee/tiny-imagenet"
ds = datasets.load_dataset(dataset_name, split=datasets.Split.TRAIN)
output_directory_path = Path(__file__).parent.joinpath("example_test_outputs", dataset_name.replace("/", "_"))
output_directory_path.mkdir(exist_ok=True, parents=True)
output_filepath = output_directory_path.joinpath("ds.parquet")
ds.to_parquet(str(output_filepath))

reloaded_ds = datasets.load_dataset(str(output_directory_path), split=datasets.Split.TRAIN)

assert ds.features == reloaded_ds.features
```

Prior to the change in this PR this script raises an `AssertionError` and the `Image` features lose their type after serialisation. After the change in this PR, the assertion does not raise an error and manual inspection of the features shows type `Image` for the respective columns of `reloaded_ds `.

Some open questions:
* How/where can I best add new unit tests for this implementation?
* What dataset would I best use in the tests? I chose `Maysee/tiny-imagenet` mainly because it is small and contains an ?Image` feature that can be used to test, but I'd be happy for suggestions on a suitable data source to use.
* Currently I'm calling `datasets.arrow_writer.ArrowWriter._build_metadata` as I need the same logic. However, I'm not happy with the coupling between `datasets.io.parquet` and `datasets.arrow_writer` it leaves me with. Suggest to factor this common logic out into a helper function and reuse it from both of these. Do you agree and if yes, could you please guide me where I would best place this function?

Many thanks in advance and kind regards,
MFreidank


  