pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function
NONE
@lhoestq 
2025-02-21T11:02:19Z
https://github.com/huggingface/datasets/issues/7418#issuecomment-2674250890
https://api.github.com/repos/huggingface/datasets/issues/7418
IC_kwDODunzps6fZdSK
https://api.github.com/repos/huggingface/datasets/issues/comments/2674250890/reactions
2025-02-21T11:02:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2674250890
https://avatars.githubusercontent.com/u/15705569?v=4
https://api.github.com/users/alexxchen/events{/privacy}
https://api.github.com/users/alexxchen/followers
https://api.github.com/users/alexxchen/following{/other_user}
https://api.github.com/users/alexxchen/gists{/gist_id}
https://github.com/alexxchen
alexxchen
MDQ6VXNlcjE1NzA1NTY5
https://api.github.com/users/alexxchen/orgs
https://api.github.com/users/alexxchen/received_events
https://api.github.com/users/alexxchen/repos
https://api.github.com/users/alexxchen/starred{/owner}{/repo}
https://api.github.com/users/alexxchen/subscriptions
User
https://api.github.com/users/alexxchen
public
MEMBER
Can you try passing text: None for the image object ? Pyarrow expects all the objects to have the exact same type, in particular the dicttionaries in "content" should all have the keys "type" and "text"
2025-02-24T10:36:58Z
https://github.com/huggingface/datasets/issues/7418#issuecomment-2678018863
https://api.github.com/repos/huggingface/datasets/issues/7418
IC_kwDODunzps6fn1Mv
https://api.github.com/repos/huggingface/datasets/issues/comments/2678018863/reactions
2025-02-24T10:36:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2678018863
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
The following modification on system prompt works, but it is different from the usual way to use it.
```
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": [{"type": "text", "text": system_prompt}]})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}
```
2025-02-24T15:41:33Z
https://github.com/huggingface/datasets/issues/7418#issuecomment-2678860387
https://api.github.com/repos/huggingface/datasets/issues/7418
IC_kwDODunzps6frCpj
https://api.github.com/repos/huggingface/datasets/issues/comments/2678860387/reactions
2025-02-24T15:41:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2678860387
https://avatars.githubusercontent.com/u/15705569?v=4
https://api.github.com/users/alexxchen/events{/privacy}
https://api.github.com/users/alexxchen/followers
https://api.github.com/users/alexxchen/following{/other_user}
https://api.github.com/users/alexxchen/gists{/gist_id}
https://github.com/alexxchen
alexxchen
MDQ6VXNlcjE1NzA1NTY5
https://api.github.com/users/alexxchen/orgs
https://api.github.com/users/alexxchen/received_events
https://api.github.com/users/alexxchen/repos
https://api.github.com/users/alexxchen/starred{/owner}{/repo}
https://api.github.com/users/alexxchen/subscriptions
User
https://api.github.com/users/alexxchen
public
MEMBER
Good to know ! But yes Arrow / Parquet have this typing limitation (which is great to ensure data integrity, but constraining at the same time). It's is really blocking you, feel free to ping the arrow team / community if they plan to have a Union type or a JSON type
2025-02-25T15:26:43Z
https://github.com/huggingface/datasets/issues/7418#issuecomment-2682370837
https://api.github.com/repos/huggingface/datasets/issues/7418
IC_kwDODunzps6f4bsV
https://api.github.com/repos/huggingface/datasets/issues/comments/2682370837/reactions
2025-02-25T15:26:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2682370837
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I encounter exactly the similar problem when using pyarrow. This issue truly helps a lot.
2025-07-11T13:06:09Z
https://github.com/huggingface/datasets/issues/7418#issuecomment-3062272890
https://api.github.com/repos/huggingface/datasets/issues/7418
IC_kwDODunzps62hpN6
https://api.github.com/repos/huggingface/datasets/issues/comments/3062272890/reactions
2025-07-11T13:06:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3062272890
https://avatars.githubusercontent.com/u/133885487?v=4
https://api.github.com/users/Wangningsen/events{/privacy}
https://api.github.com/users/Wangningsen/followers
https://api.github.com/users/Wangningsen/following{/other_user}
https://api.github.com/users/Wangningsen/gists{/gist_id}
https://github.com/Wangningsen
Wangningsen
U_kgDOB_ruLw
https://api.github.com/users/Wangningsen/orgs
https://api.github.com/users/Wangningsen/received_events
https://api.github.com/users/Wangningsen/repos
https://api.github.com/users/Wangningsen/starred{/owner}{/repo}
https://api.github.com/users/Wangningsen/subscriptions
User
https://api.github.com/users/Wangningsen
public
### Describe the bug

Encounter pyarrow.lib.arrowinvalid error with map function in some example when loading the dataset

### Steps to reproduce the bug

```
from datasets import load_dataset
from PIL import Image, PngImagePlugin

dataset = load_dataset("leonardPKU/GEOQA_R1V_Train_8K")
system_prompt="You are a helpful AI Assistant"
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": system_prompt})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}

def check_data_types(example):
    for key, value in example.items():
        if key == 'image':
            if not isinstance(value, PngImagePlugin.PngImageFile):
                print(value)
        if key == "problem" or key == "solution":
            if not isinstance(value, str):
                print(value)

    return example

dataset = dataset.map(check_data_types)
dataset = dataset.map(make_conversation)
```

### Expected behavior

Successfully process the dataset with map

### Environment info

datasets==3.3.1