Image Encoding Issue when submitting a Parquet Dataset
CONTRIBUTOR
Hi @PhilippeMoussalli thanks for opening a detailed issue. It seems the issue is more related to the `datasets` library so I'll ping @lhoestq @mariosasko on this one :) 

(edit: also can one of you move the issue to the datasets repo? Thanks in advance 🙏)
2023-05-16T10:13:40Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1549629660
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5cXXjc
https://api.github.com/repos/huggingface/datasets/issues/comments/1549629660/reactions
2023-05-16T10:14:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549629660
https://avatars.githubusercontent.com/u/11801849?v=4
https://api.github.com/users/Wauplin/events{/privacy}
https://api.github.com/users/Wauplin/followers
https://api.github.com/users/Wauplin/following{/other_user}
https://api.github.com/users/Wauplin/gists{/gist_id}
https://github.com/Wauplin
Wauplin
MDQ6VXNlcjExODAxODQ5
https://api.github.com/users/Wauplin/orgs
https://api.github.com/users/Wauplin/received_events
https://api.github.com/users/Wauplin/repos
https://api.github.com/users/Wauplin/starred{/owner}{/repo}
https://api.github.com/users/Wauplin/subscriptions
User
https://api.github.com/users/Wauplin
public
MEMBER
Hi ! The `Image()` info is stored in the **schema metadata**. More precisely there should be a "huggingface" field in the schema metadata that contains the `datasets` feature type of each column.

To fix your issue, you can use the same schema as the original Parquet files to write the new ones. You can also get the schema with metadata from a `Features` object, e.g.

```python
from datasets import Features, Image, Value

features = Features({"image": Image(), "text": Value("string")})
schema = features.arrow_schema
print(schema.metadata)
# {b'huggingface': b'{"info": {"features": {"image": {"_type": "Image"}, "text": {"dtype": "string", "_type": "Value"}}}}'}
```
2023-05-16T13:10:29Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1549645951
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5cXbh_
https://api.github.com/repos/huggingface/datasets/issues/comments/1549645951/reactions
2023-05-16T13:10:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549645951
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It appears that the parquet files at `hf://datasets/lambdalabs/pokemon-blip-captions` don't have this metadata, and it is defined in the dataset_infos.json instead (legacy).

You can get the right schema with the HF metadata this way:

```python
from datasets import load_dataset_builder

features = load_dataset_builder("lambdalabs/pokemon-blip-captions").info.features
schema = features.arrow_schema
```
2023-05-16T13:16:26Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1549659072
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5cXevA
https://api.github.com/repos/huggingface/datasets/issues/comments/1549659072/reactions
2023-05-16T13:16:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549659072
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Btw in the future we might add support for an dedicated Image extension type in Arrow so that you won't need to add the schema metadata anymore ;)
2023-05-16T13:19:47Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1549664464
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5cXgDQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1549664464/reactions
2023-05-16T13:19:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549664464
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks @Wauplin @lhoestq for the quick reply :)! 

I tried your approach by passing the huggingface schema to the dask writer 

```
from datasets import Features, Image, Value
df = dd.read_parquet(f"hf://datasets/lambdalabs/pokemon-blip-captions",index=False)
features = Features({"image": Image(), "text": Value("string")})
schema = features.arrow_schema
dd.to_parquet(df, path = "hf://datasets/philippemo/dummy_dataset/data", schema=schema)
```
At first it didn't work as I was not able to visualize the images, so then I manually added the `dataset_infos.json` from the example dataset and it worked :)

However, It's not very ideal since there are some metadata in that file that need to be computed in order to load the data properly such as `num_of_bytes` and `num_examples` which might be unknown in my use case. 

![Screenshot from 2023-05-16 16-54-55](https://github.com/huggingface/datasets/assets/47530815/b2b448d2-d3d8-43a7-9682-9c0187a5192b)

Do you have any pointers there? you mentioned that `datasets_info.json` will be deprecated/legacy. Could you point me to some example image datasets on the hub that are stored as parquet and don't have the `datasets_info.json`?


2023-05-16T15:08:14Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1549861473
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5cYQJh
https://api.github.com/repos/huggingface/datasets/issues/comments/1549861473/reactions
2023-05-16T15:08:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549861473
https://avatars.githubusercontent.com/u/47530815?v=4
https://api.github.com/users/PhilippeMoussalli/events{/privacy}
https://api.github.com/users/PhilippeMoussalli/followers
https://api.github.com/users/PhilippeMoussalli/following{/other_user}
https://api.github.com/users/PhilippeMoussalli/gists{/gist_id}
https://github.com/PhilippeMoussalli
PhilippeMoussalli
MDQ6VXNlcjQ3NTMwODE1
https://api.github.com/users/PhilippeMoussalli/orgs
https://api.github.com/users/PhilippeMoussalli/received_events
https://api.github.com/users/PhilippeMoussalli/repos
https://api.github.com/users/PhilippeMoussalli/starred{/owner}{/repo}
https://api.github.com/users/PhilippeMoussalli/subscriptions
User
https://api.github.com/users/PhilippeMoussalli
public
MEMBER
You don't need the dataset_infos.json file as long as you have the schema with HF metadata ;)
I could also check that it works fine myself on the git revision without the dataset_infos.json file.

What made you think it didn't work ?
2023-05-17T18:47:18Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1551892056
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5cf_5Y
https://api.github.com/repos/huggingface/datasets/issues/comments/1551892056/reactions
2023-05-17T18:47:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1551892056
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> You don't need the dataset_infos.json file as long as you have the schema with HF metadata ;) I could also check that it works fine myself on the git revision without the dataset_infos.json file.
> 
> What made you think it didn't work ?

Those are two identical dataset repos where both were pushed with dask with the specified schema you mentioned above. I then uploaded the `dataset_infos.json` manually taken from the original example dataset into one of them. 

* **With schema**: https://huggingface.co/datasets/philippemo/dummy_dataset_with_schema
* **Without schema**: https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema

You can see that in the examples without schema the images fail to render properly. When loaded with `datasets` they return an dict and not a Pillow Image 
2023-05-22T09:54:06Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1556912438
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5czJk2
https://api.github.com/repos/huggingface/datasets/issues/comments/1556912438/reactions
2023-05-22T09:54:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1556912438
https://avatars.githubusercontent.com/u/47530815?v=4
https://api.github.com/users/PhilippeMoussalli/events{/privacy}
https://api.github.com/users/PhilippeMoussalli/followers
https://api.github.com/users/PhilippeMoussalli/following{/other_user}
https://api.github.com/users/PhilippeMoussalli/gists{/gist_id}
https://github.com/PhilippeMoussalli
PhilippeMoussalli
MDQ6VXNlcjQ3NTMwODE1
https://api.github.com/users/PhilippeMoussalli/orgs
https://api.github.com/users/PhilippeMoussalli/received_events
https://api.github.com/users/PhilippeMoussalli/repos
https://api.github.com/users/PhilippeMoussalli/starred{/owner}{/repo}
https://api.github.com/users/PhilippeMoussalli/subscriptions
User
https://api.github.com/users/PhilippeMoussalli
public
MEMBER
I see ! I think it's a bug on our side - it should work without the metadata - let me investigate
2023-05-25T14:14:21Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1562987595
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5dKUxL
https://api.github.com/repos/huggingface/datasets/issues/comments/1562987595/reactions
2023-05-25T14:14:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1562987595
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright, it's fixed: https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema

It shows the image correctly now - even without the extra metadata :)
2023-05-25T14:18:14Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1562993874
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5dKWTS
https://api.github.com/repos/huggingface/datasets/issues/comments/1562993874/reactions
2023-05-25T14:18:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1562993874
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks @lhoestq! 
I tested pushing a dataset again without the metadata and it works perfectly! 
I appreciate the help
2023-05-30T14:17:08Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1568517866
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5dfa7q
https://api.github.com/repos/huggingface/datasets/issues/comments/1568517866/reactions
2023-05-30T14:17:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1568517866
https://avatars.githubusercontent.com/u/47530815?v=4
https://api.github.com/users/PhilippeMoussalli/events{/privacy}
https://api.github.com/users/PhilippeMoussalli/followers
https://api.github.com/users/PhilippeMoussalli/following{/other_user}
https://api.github.com/users/PhilippeMoussalli/gists{/gist_id}
https://github.com/PhilippeMoussalli
PhilippeMoussalli
MDQ6VXNlcjQ3NTMwODE1
https://api.github.com/users/PhilippeMoussalli/orgs
https://api.github.com/users/PhilippeMoussalli/received_events
https://api.github.com/users/PhilippeMoussalli/repos
https://api.github.com/users/PhilippeMoussalli/starred{/owner}{/repo}
https://api.github.com/users/PhilippeMoussalli/subscriptions
User
https://api.github.com/users/PhilippeMoussalli
public
NONE
Hi @lhoestq, 

I'v tried pushing another dataset again and I think the issue reappeared again: 

```
df = dd.read_parquet(f"hf://datasets/lambdalabs/pokemon-blip-captions")
features = datasets.Features({"image": datasets.Image(), "text": datasets.Value("string")})
schema = features.arrow_schema
dd.to_parquet(df, path = "hf://datasets/philippemo/dummy_dataset_without_schema_12_06/data", schema=schema)
```

Here is the dataset:  
 https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema_12_06
The one that was working 2 weeks ago still seems to be intact though, it might be that It rendered properly when it was initially submitted and after this something was reverted from your side:
https://huggingface.co/datasets/philippemo/dummy_dataset_without_schema

It's weird because nothing really changed from the implementation, might be another issue in the hub backend. Do you have any pointers on how to resolve this? 
2023-06-12T15:12:48Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1587543818
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5en_8K
https://api.github.com/repos/huggingface/datasets/issues/comments/1587543818/reactions
2023-06-12T15:12:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1587543818
https://avatars.githubusercontent.com/u/47530815?v=4
https://api.github.com/users/PhilippeMoussalli/events{/privacy}
https://api.github.com/users/PhilippeMoussalli/followers
https://api.github.com/users/PhilippeMoussalli/following{/other_user}
https://api.github.com/users/PhilippeMoussalli/gists{/gist_id}
https://github.com/PhilippeMoussalli
PhilippeMoussalli
MDQ6VXNlcjQ3NTMwODE1
https://api.github.com/users/PhilippeMoussalli/orgs
https://api.github.com/users/PhilippeMoussalli/received_events
https://api.github.com/users/PhilippeMoussalli/repos
https://api.github.com/users/PhilippeMoussalli/starred{/owner}{/repo}
https://api.github.com/users/PhilippeMoussalli/subscriptions
User
https://api.github.com/users/PhilippeMoussalli
public
MEMBER
We're doing some changes in the way we're handling image parquet datasets right now. We'll include the fix from https://github.com/huggingface/datasets/pull/5921 in the new datasets-server version in the coming days
2023-06-12T15:16:55Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1587551425
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5eoBzB
https://api.github.com/repos/huggingface/datasets/issues/comments/1587551425/reactions
2023-06-12T15:16:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1587551425
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
alright thanks for the update :), would that be part of the new release of datasets or is it something separate? if so, where can I track it? 
2023-06-13T07:10:15Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1588678648
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5esU_4
https://api.github.com/repos/huggingface/datasets/issues/comments/1588678648/reactions
2023-06-13T07:10:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1588678648
https://avatars.githubusercontent.com/u/47530815?v=4
https://api.github.com/users/PhilippeMoussalli/events{/privacy}
https://api.github.com/users/PhilippeMoussalli/followers
https://api.github.com/users/PhilippeMoussalli/following{/other_user}
https://api.github.com/users/PhilippeMoussalli/gists{/gist_id}
https://github.com/PhilippeMoussalli
PhilippeMoussalli
MDQ6VXNlcjQ3NTMwODE1
https://api.github.com/users/PhilippeMoussalli/orgs
https://api.github.com/users/PhilippeMoussalli/received_events
https://api.github.com/users/PhilippeMoussalli/repos
https://api.github.com/users/PhilippeMoussalli/starred{/owner}{/repo}
https://api.github.com/users/PhilippeMoussalli/subscriptions
User
https://api.github.com/users/PhilippeMoussalli
public
MEMBER
Once the new version of `datasets` is released (tomorrow probably) we'll open an issue on https://github.com/huggingface/datasets-server to update to this version :)
2023-06-13T16:24:31Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1589642612
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5ewAV0
https://api.github.com/repos/huggingface/datasets/issues/comments/1589642612/reactions
2023-06-13T16:24:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1589642612
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright we did the update :) This is fixed for good now
2023-06-16T09:28:57Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1594394100
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5fCIX0
https://api.github.com/repos/huggingface/datasets/issues/comments/1594394100/reactions
2023-06-16T09:28:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1594394100
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes thanks 🎉🎉🎉
2023-06-16T12:48:38Z
https://github.com/huggingface/datasets/issues/5869#issuecomment-1594624636
https://api.github.com/repos/huggingface/datasets/issues/5869
IC_kwDODunzps5fDAp8
https://api.github.com/repos/huggingface/datasets/issues/comments/1594624636/reactions
2023-06-16T12:48:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1594624636
https://avatars.githubusercontent.com/u/47530815?v=4
https://api.github.com/users/PhilippeMoussalli/events{/privacy}
https://api.github.com/users/PhilippeMoussalli/followers
https://api.github.com/users/PhilippeMoussalli/following{/other_user}
https://api.github.com/users/PhilippeMoussalli/gists{/gist_id}
https://github.com/PhilippeMoussalli
PhilippeMoussalli
MDQ6VXNlcjQ3NTMwODE1
https://api.github.com/users/PhilippeMoussalli/orgs
https://api.github.com/users/PhilippeMoussalli/received_events
https://api.github.com/users/PhilippeMoussalli/repos
https://api.github.com/users/PhilippeMoussalli/starred{/owner}{/repo}
https://api.github.com/users/PhilippeMoussalli/subscriptions
User
https://api.github.com/users/PhilippeMoussalli
public
### Describe the bug

Hello,

I'd like to report an issue related to pushing a dataset represented as a Parquet file to a dataset repository using Dask. Here are the details:

We attempted to load an example dataset in Parquet format from the Hugging Face (HF) filesystem using Dask with the following code snippet:
```
import dask.dataframe as dd
df = dd.read_parquet("hf://datasets/lambdalabs/pokemon-blip-captions",index=False)
```
In this dataset, the "image" column is represented as a dictionary/struct with the format:

```
df = df.compute()
df["image"].iloc[0].keys()
-> dict_keys(['bytes', 'path'])
```
I think this is the format encoded by the [`Image`](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Image) feature extractor from datasets to format suitable for Arrow. 

The next step was to push the dataset to a repository that I created:
```
dd.to_parquet(dask_df, path = "hf://datasets/philippemo/dummy_dataset/data")
```

However, after pushing the dataset using Dask, the "image" column is now represented as the encoded dictionary `(['bytes', 'path'])`, and the images are not properly visualized. You can find the dataset here: [Link to the problematic dataset](https://huggingface.co/datasets/philippemo/dummy_dataset).

It's worth noting that both the original dataset and the one submitted with Dask have the same schema with minor alterations related to metadata:

**[ Schema of original dummy example.](https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions/blob/main/data/train-00000-of-00001-566cc9b19d7203f8.parquet)**   
```
image: struct<bytes: binary, path: null>
  child 0, bytes: binary
  child 1, path: null
text: string
```
**[ Schema of pushed dataset with dask](https://huggingface.co/datasets/philippemo/dummy_dataset/blob/main/data/part.0.parquet)**
```
image: struct<bytes: binary, path: null>
  child 0, bytes: binary
  child 1, path: null
text: string
```

This issue seems to be related to an encoding type that occurs when pushing a model to the hub. Normally, models should be represented as an HF dataset before pushing, but we are working with an example where we need to push large datasets using Dask.

Could you please provide clarification on how to resolve this issue?

Thank you!


### Reproduction

To get the schema I downloaded the parquet files and used pyarrow.parquet to read the schema
```
import pyarrow.parquet
pyarrow.parquet.read_schema(<path_to_parquet>, memory_map=True)
```

### Logs

_No response_

### System info

```shell
- huggingface_hub version: 0.14.1
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/philippe/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: philippemo
- Configured git credential helpers: cache
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.4.0
- hf_transfer: N/A
- gradio: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/philippe/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/philippe/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/philippe/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
```
