ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset
CONTRIBUTOR
@mariosasko I'm getting a similar issue when creating a Dataset from a Pandas dataframe, like so:

```
from datasets import Dataset, Features, Image, Value
import pandas as pd
import requests
import PIL

# we need to define the features ourselves
features = Features({
    'a': Value(dtype='int32'),
    'b': Image(),
})

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = PIL.Image.open(requests.get(url, stream=True).raw)

df = pd.DataFrame({"a": [1, 2], 
                   "b": [image, image]})

dataset = Dataset.from_pandas(df, features=features) 
```
results in 

```
ArrowInvalid: ('Could not convert <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480 at 0x7F7991A15C10> with type JpegImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column b with type object')
```

Will the PR linked above also fix that?
2022-08-12T13:28:34Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1213110543
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5ITpkP
https://api.github.com/repos/huggingface/datasets/issues/comments/1213110543/reactions
2022-08-12T13:30:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213110543
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
COLLABORATOR
I would expect this to work, but it doesn't. Shouldn't be too hard to fix tho (in a subsequent PR).
2022-08-12T14:57:06Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1213206077
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5IUA49
https://api.github.com/repos/huggingface/datasets/issues/comments/1213206077/reactions
2022-08-12T14:57:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213206077
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi @mariosasko just wanted to check in if there is a PR to follow for this. I was looking to create a demo app using this. If it's not working I can just use byte encoded images in the dataset which are not displayed. 
2022-08-19T12:09:51Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1220600508
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5IwOK8
https://api.github.com/repos/huggingface/datasets/issues/comments/1220600508/reactions
2022-08-19T12:09:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1220600508
https://avatars.githubusercontent.com/u/7209588?v=4
https://api.github.com/users/darraghdog/events{/privacy}
https://api.github.com/users/darraghdog/followers
https://api.github.com/users/darraghdog/following{/other_user}
https://api.github.com/users/darraghdog/gists{/gist_id}
https://github.com/darraghdog
darraghdog
MDQ6VXNlcjcyMDk1ODg=
https://api.github.com/users/darraghdog/orgs
https://api.github.com/users/darraghdog/received_events
https://api.github.com/users/darraghdog/repos
https://api.github.com/users/darraghdog/starred{/owner}{/repo}
https://api.github.com/users/darraghdog/subscriptions
User
https://api.github.com/users/darraghdog
public
COLLABORATOR
Hi @darraghdog! No PR yet, but I plan to fix this before the next release.
2022-08-19T12:41:29Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1220629791
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5IwVUf
https://api.github.com/repos/huggingface/datasets/issues/comments/1220629791/reactions
2022-08-19T12:41:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1220629791
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
I was just pointed here by @mariosasko, meanwhile I found a workaround using `encode_example` like so:

```
from datasets import load_from_disk, Dataset
DATASET_PATH = "/hf/m4-master/data/cm4/cm4-10000-v0.1"
ds1 = load_from_disk(DATASET_PATH)
ds2 = Dataset.from_dict(mapping={k: [] for k in ds1[99].keys()},
                       features=ds1.features
)
for i in range(2):
    # could add several representative items here
    row = ds1[99]
    row_encoded = ds2.features.encode_example(row)
    ds2 = ds2.add_item(row_encoded)
```
2022-09-20T15:49:14Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1252554689
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5KqHfB
https://api.github.com/repos/huggingface/datasets/issues/comments/1252554689/reactions
2022-09-20T15:49:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1252554689
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
CONTRIBUTOR
Hmm, interesting. If I create the dataset on the fly:

```
from datasets import load_from_disk, Dataset
DATASET_PATH = "/hf/m4-master/data/cm4/cm4-10000-v0.1"
ds1 = load_from_disk(DATASET_PATH)
ds2 = Dataset.from_dict(mapping={k: [v]*2 for k, v in ds1[99].items()},
                        features=ds1.features)
```

it doesn't fail with the error in the OP, as `from_dict` performs `encode_batch`.

However if I try to use this dataset it fails now with:

```
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 524, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2775, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2655, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2347, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "debug_leak2.py", line 235, in split_pack_and_pad
    images.append(image_transform(image.convert("RGB")))
AttributeError: 'dict' object has no attribute 'convert'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "debug_leak2.py", line 418, in <module>
    train_loader, val_loader = get_dataloaders()
  File "debug_leak2.py", line 348, in get_dataloaders
    dataset = dataset.map(mapper, batch_size=32, batched=True, remove_columns=dataset.column_names, num_proc=4)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2500, in map
    transformed_shards[index] = async_result.get()
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/multiprocess/pool.py", line 771, in get
    raise self._value
AttributeError: 'dict' object has no attribute 'convert'
```

but if I create that same dataset one item at a time as in the previous comment's code snippet it doesn't fail.

The features of this dataset are set to:

```
{'texts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 
'images': Sequence(feature=Image(decode=True, id=None), length=-1, id=None)}
```
2022-09-20T17:16:13Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1252660340
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5KqhR0
https://api.github.com/repos/huggingface/datasets/issues/comments/1252660340/reactions
2022-09-20T17:17:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1252660340
https://avatars.githubusercontent.com/u/10676103?v=4
https://api.github.com/users/stas00/events{/privacy}
https://api.github.com/users/stas00/followers
https://api.github.com/users/stas00/following{/other_user}
https://api.github.com/users/stas00/gists{/gist_id}
https://github.com/stas00
stas00
MDQ6VXNlcjEwNjc2MTAz
https://api.github.com/users/stas00/orgs
https://api.github.com/users/stas00/received_events
https://api.github.com/users/stas00/repos
https://api.github.com/users/stas00/starred{/owner}{/repo}
https://api.github.com/users/stas00/subscriptions
User
https://api.github.com/users/stas00
public
NONE
> @mariosasko I'm getting a similar issue when creating a Dataset from a Pandas dataframe, like so:
> 
> ```
> from datasets import Dataset, Features, Image, Value
> import pandas as pd
> import requests
> import PIL
> 
> # we need to define the features ourselves
> features = Features({
>     'a': Value(dtype='int32'),
>     'b': Image(),
> })
> 
> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
> image = PIL.Image.open(requests.get(url, stream=True).raw)
> 
> df = pd.DataFrame({"a": [1, 2], 
>                    "b": [image, image]})
> 
> dataset = Dataset.from_pandas(df, features=features) 
> ```
> 
> results in
> 
> ```
> ArrowInvalid: ('Could not convert <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480 at 0x7F7991A15C10> with type JpegImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column b with type object')
> ```
> 
> Will the PR linked above also fix that?

It looks like the problem still exists.
Any news ? Any good workaround ?

Thank you
2023-02-12T20:08:06Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1427121036
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5VECOM
https://api.github.com/repos/huggingface/datasets/issues/comments/1427121036/reactions
2023-02-12T20:08:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1427121036
https://avatars.githubusercontent.com/u/2767433?v=4
https://api.github.com/users/MaxxTr/events{/privacy}
https://api.github.com/users/MaxxTr/followers
https://api.github.com/users/MaxxTr/following{/other_user}
https://api.github.com/users/MaxxTr/gists{/gist_id}
https://github.com/MaxxTr
MaxxTr
MDQ6VXNlcjI3Njc0MzM=
https://api.github.com/users/MaxxTr/orgs
https://api.github.com/users/MaxxTr/received_events
https://api.github.com/users/MaxxTr/repos
https://api.github.com/users/MaxxTr/starred{/owner}{/repo}
https://api.github.com/users/MaxxTr/subscriptions
User
https://api.github.com/users/MaxxTr
public
NONE
There is a workaround: 
Create a loader python scrypt and upload the dataset to huggingface.

Here is an example how to do that:

https://huggingface.co/datasets/jamescalam/image-text-demo/tree/main

and Here are videos with explanations:

https://www.youtube.com/watch?v=lqK4ocAKveE and https://www.youtube.com/watch?v=ODdKC30dT8c
2023-02-13T11:25:27Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1427781086
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5VGjXe
https://api.github.com/repos/huggingface/datasets/issues/comments/1427781086/reactions
2023-02-13T11:25:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1427781086
https://avatars.githubusercontent.com/u/2767433?v=4
https://api.github.com/users/MaxxTr/events{/privacy}
https://api.github.com/users/MaxxTr/followers
https://api.github.com/users/MaxxTr/following{/other_user}
https://api.github.com/users/MaxxTr/gists{/gist_id}
https://github.com/MaxxTr
MaxxTr
MDQ6VXNlcjI3Njc0MzM=
https://api.github.com/users/MaxxTr/orgs
https://api.github.com/users/MaxxTr/received_events
https://api.github.com/users/MaxxTr/repos
https://api.github.com/users/MaxxTr/starred{/owner}{/repo}
https://api.github.com/users/MaxxTr/subscriptions
User
https://api.github.com/users/MaxxTr
public
CONTRIBUTOR
cc @mariosasko gentle ping for a fix :)
2023-02-15T09:24:01Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1431003740
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5VS2Jc
https://api.github.com/repos/huggingface/datasets/issues/comments/1431003740/reactions
2023-02-15T09:24:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1431003740
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
Any update on this? I'm still facing this issure. Any workaround?
2023-03-31T17:40:35Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1492352844
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5Y839M
https://api.github.com/repos/huggingface/datasets/issues/comments/1492352844/reactions
2023-03-31T17:40:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1492352844
https://avatars.githubusercontent.com/u/12396148?v=4
https://api.github.com/users/chumpblocckami/events{/privacy}
https://api.github.com/users/chumpblocckami/followers
https://api.github.com/users/chumpblocckami/following{/other_user}
https://api.github.com/users/chumpblocckami/gists{/gist_id}
https://github.com/chumpblocckami
chumpblocckami
MDQ6VXNlcjEyMzk2MTQ4
https://api.github.com/users/chumpblocckami/orgs
https://api.github.com/users/chumpblocckami/received_events
https://api.github.com/users/chumpblocckami/repos
https://api.github.com/users/chumpblocckami/starred{/owner}{/repo}
https://api.github.com/users/chumpblocckami/subscriptions
User
https://api.github.com/users/chumpblocckami
public
NONE
I was facing the same issue. Downgrading datasets from 2.11.0 to 2.4.0 solved the issue. 
2023-04-11T16:04:34Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1503681568
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5ZoFwg
https://api.github.com/repos/huggingface/datasets/issues/comments/1503681568/reactions
2023-04-11T16:04:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1503681568
https://avatars.githubusercontent.com/u/25868753?v=4
https://api.github.com/users/umarpreet1/events{/privacy}
https://api.github.com/users/umarpreet1/followers
https://api.github.com/users/umarpreet1/following{/other_user}
https://api.github.com/users/umarpreet1/gists{/gist_id}
https://github.com/umarpreet1
umarpreet1
MDQ6VXNlcjI1ODY4NzUz
https://api.github.com/users/umarpreet1/orgs
https://api.github.com/users/umarpreet1/received_events
https://api.github.com/users/umarpreet1/repos
https://api.github.com/users/umarpreet1/starred{/owner}{/repo}
https://api.github.com/users/umarpreet1/subscriptions
User
https://api.github.com/users/umarpreet1
public
NONE
> Any update on this? I'm still facing this issure. Any workaround?

I was able to resolve my issue with a quick workaround: 

```
from collections import defaultdict
from datasets import Dataset
  
data = defaultdict(list)
for idx in tqdm(range( len(dataloader)),desc="Captioning..."):
    img = dataloader[idx]
    data['image'].append(img)
    data['text'].append(f"{img_{idx}})
  
dataset = Dataset.from_dict(data)
dataset = dataset.filter(lambda example: example['image'] is not None)
dataset = dataset.filter(lambda example: example['text'] is not None)
  
dataset.push_to_hub(path-to-repo', private=False)
```

Hope it helps!
Happy coding
2023-04-13T13:38:30Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1506980854
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5Z0rP2
https://api.github.com/repos/huggingface/datasets/issues/comments/1506980854/reactions
2023-04-13T13:39:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1506980854
https://avatars.githubusercontent.com/u/12396148?v=4
https://api.github.com/users/chumpblocckami/events{/privacy}
https://api.github.com/users/chumpblocckami/followers
https://api.github.com/users/chumpblocckami/following{/other_user}
https://api.github.com/users/chumpblocckami/gists{/gist_id}
https://github.com/chumpblocckami
chumpblocckami
MDQ6VXNlcjEyMzk2MTQ4
https://api.github.com/users/chumpblocckami/orgs
https://api.github.com/users/chumpblocckami/received_events
https://api.github.com/users/chumpblocckami/repos
https://api.github.com/users/chumpblocckami/starred{/owner}{/repo}
https://api.github.com/users/chumpblocckami/subscriptions
User
https://api.github.com/users/chumpblocckami
public
NONE
> > Any update on this? I'm still facing this issure. Any workaround?
> 
> I was able to resolve my issue with a quick workaround:
> 
> ```
> from collections import defaultdict
> from datasets import Dataset
>   
> data = defaultdict(list)
> for idx in tqdm(range( len(dataloader)),desc="Captioning..."):
>     img = dataloader[idx]
>     data['image'].append(img)
>     data['text'].append(f"{img_{idx}})
>   
> dataset = Dataset.from_dict(data)
> dataset = dataset.filter(lambda example: example['image'] is not None)
> dataset = dataset.filter(lambda example: example['text'] is not None)
>   
> dataset.push_to_hub(path-to-repo', private=False)
> ```
> 
> Hope it helps! Happy coding

It works!! 
2023-04-26T11:20:25Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1523250540
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5ayvVs
https://api.github.com/repos/huggingface/datasets/issues/comments/1523250540/reactions
2023-04-26T11:20:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1523250540
https://avatars.githubusercontent.com/u/126648805?v=4
https://api.github.com/users/thinh-huynh-re/events{/privacy}
https://api.github.com/users/thinh-huynh-re/followers
https://api.github.com/users/thinh-huynh-re/following{/other_user}
https://api.github.com/users/thinh-huynh-re/gists{/gist_id}
https://github.com/thinh-huynh-re
thinh-huynh-re
U_kgDOB4yB5Q
https://api.github.com/users/thinh-huynh-re/orgs
https://api.github.com/users/thinh-huynh-re/received_events
https://api.github.com/users/thinh-huynh-re/repos
https://api.github.com/users/thinh-huynh-re/starred{/owner}{/repo}
https://api.github.com/users/thinh-huynh-re/subscriptions
User
https://api.github.com/users/thinh-huynh-re
public
NONE
> 

how did this work,  how to use this script or where to paste it?
2023-05-31T14:05:34Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-1570306056
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps5dmPgI
https://api.github.com/repos/huggingface/datasets/issues/comments/1570306056/reactions
2023-05-31T14:05:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1570306056
https://avatars.githubusercontent.com/u/125782753?v=4
https://api.github.com/users/LanceGao97/events{/privacy}
https://api.github.com/users/LanceGao97/followers
https://api.github.com/users/LanceGao97/following{/other_user}
https://api.github.com/users/LanceGao97/gists{/gist_id}
https://github.com/LanceGao97
LanceGao97
U_kgDOB39K4Q
https://api.github.com/users/LanceGao97/orgs
https://api.github.com/users/LanceGao97/received_events
https://api.github.com/users/LanceGao97/repos
https://api.github.com/users/LanceGao97/starred{/owner}{/repo}
https://api.github.com/users/LanceGao97/subscriptions
User
https://api.github.com/users/LanceGao97
public
NONE
I had a similar issue to @NielsRogge where I was unable to create a dataset from a Pandas DataFrame containing PIL.Images.

I found another workaround that works in this case which involves converting the DataFrame to a python dictionary, and then creating a dataset from said python dictionary.

This is a generic example of my workaround. The example assumes that you have your data in a Pandas DataFrame variable called "dataframe" plus a dictionary of your data's features in a variable called "features".
```
import datasets

dictionary = dataframe.to_dict(orient='list')
dataset = datasets.Dataset.from_dict(dictionary, features=features)
```
2024-04-11T02:23:07Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-2048822155
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps56HouL
https://api.github.com/repos/huggingface/datasets/issues/comments/2048822155/reactions
2024-04-11T02:23:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2048822155
https://avatars.githubusercontent.com/u/22741414?v=4
https://api.github.com/users/Ekhao/events{/privacy}
https://api.github.com/users/Ekhao/followers
https://api.github.com/users/Ekhao/following{/other_user}
https://api.github.com/users/Ekhao/gists{/gist_id}
https://github.com/Ekhao
Ekhao
MDQ6VXNlcjIyNzQxNDE0
https://api.github.com/users/Ekhao/orgs
https://api.github.com/users/Ekhao/received_events
https://api.github.com/users/Ekhao/repos
https://api.github.com/users/Ekhao/starred{/owner}{/repo}
https://api.github.com/users/Ekhao/subscriptions
User
https://api.github.com/users/Ekhao
public
CONTRIBUTOR
cc @mariosasko this issue has been open for 2 years, would be great to resolve it :)
2024-04-11T06:59:31Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-2049043321
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps56Iet5
https://api.github.com/repos/huggingface/datasets/issues/comments/2049043321/reactions
2024-04-11T06:59:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2049043321
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
I have the same issue, my current workaround is saving the dataframe to a csv and then loading the dataset from the csv. Would also appreciate it a fix :)
2024-04-21T19:28:10Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-2068173639
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps57RdNH
https://api.github.com/repos/huggingface/datasets/issues/comments/2068173639/reactions
2024-04-21T19:28:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2068173639
https://avatars.githubusercontent.com/u/22015936?v=4
https://api.github.com/users/tanyav2/events{/privacy}
https://api.github.com/users/tanyav2/followers
https://api.github.com/users/tanyav2/following{/other_user}
https://api.github.com/users/tanyav2/gists{/gist_id}
https://github.com/tanyav2
tanyav2
MDQ6VXNlcjIyMDE1OTM2
https://api.github.com/users/tanyav2/orgs
https://api.github.com/users/tanyav2/received_events
https://api.github.com/users/tanyav2/repos
https://api.github.com/users/tanyav2/starred{/owner}{/repo}
https://api.github.com/users/tanyav2/subscriptions
User
https://api.github.com/users/tanyav2
public
NONE
> data = defaultdict(list)

awesome, it really works~
2024-11-25T09:26:30Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-2497395415
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps6U2zrX
https://api.github.com/repos/huggingface/datasets/issues/comments/2497395415/reactions
2024-11-25T09:26:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2497395415
https://avatars.githubusercontent.com/u/32154029?v=4
https://api.github.com/users/hunter2009pf/events{/privacy}
https://api.github.com/users/hunter2009pf/followers
https://api.github.com/users/hunter2009pf/following{/other_user}
https://api.github.com/users/hunter2009pf/gists{/gist_id}
https://github.com/hunter2009pf
hunter2009pf
MDQ6VXNlcjMyMTU0MDI5
https://api.github.com/users/hunter2009pf/orgs
https://api.github.com/users/hunter2009pf/received_events
https://api.github.com/users/hunter2009pf/repos
https://api.github.com/users/hunter2009pf/starred{/owner}{/repo}
https://api.github.com/users/hunter2009pf/subscriptions
User
https://api.github.com/users/hunter2009pf
public
NONE
I found something that can be used as solution.

I have the same problem when I've try to load the images from a pamdas dataset

If you have all on a pandas dataset try 
Dataset.from_dict( your_df.reset_index(drop=True).to_dict(orient='list'), split=set_your_split)

And this avoid the error
2024-11-29T16:35:15Z
https://github.com/huggingface/datasets/issues/4796#issuecomment-2508125036
https://api.github.com/repos/huggingface/datasets/issues/4796
IC_kwDODunzps6VfvNs
https://api.github.com/repos/huggingface/datasets/issues/comments/2508125036/reactions
2024-11-29T16:35:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2508125036
https://avatars.githubusercontent.com/u/56119719?v=4
https://api.github.com/users/juanxo90/events{/privacy}
https://api.github.com/users/juanxo90/followers
https://api.github.com/users/juanxo90/following{/other_user}
https://api.github.com/users/juanxo90/gists{/gist_id}
https://github.com/juanxo90
juanxo90
MDQ6VXNlcjU2MTE5NzE5
https://api.github.com/users/juanxo90/orgs
https://api.github.com/users/juanxo90/received_events
https://api.github.com/users/juanxo90/repos
https://api.github.com/users/juanxo90/starred{/owner}{/repo}
https://api.github.com/users/juanxo90/subscriptions
User
https://api.github.com/users/juanxo90
public
## Describe the bug

When adding a Pillow image to an existing Dataset on the hub, `add_item` fails due to the Pillow image not being automatically converted into the Image feature.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from PIL import Image

dataset = load_dataset("hf-internal-testing/example-documents")

# load any random Pillow image
image = Image.open("/content/cord_example.png").convert("RGB")

new_image = {'image': image}
dataset['test'] = dataset['test'].add_item(new_image)
```

## Expected results
The image should be automatically casted to the Image feature when using `add_item`. For now, this can be fixed by using `encode_example`:

```
import datasets

feature = datasets.Image(decode=False)
new_image = {'image': feature.encode_example(image)}
dataset['test'] = dataset['test'].add_item(new_image)
```

## Actual results

```
ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=576x864 at 0x7F7CCC4589D0> with type Image: did not recognize Python value type when inferring an Arrow data type
```
