[Docs] map() example uses undefined `tokenizer` — causes NameError
NONE
I've submitted PR #7704 which adds documentation to clarify the behavior of `map()` when returning `None`.
2025-07-27T05:39:18Z
https://github.com/huggingface/datasets/issues/7703#issuecomment-3124002704
https://api.github.com/repos/huggingface/datasets/issues/7703
IC_kwDODunzps66NH-Q
https://api.github.com/repos/huggingface/datasets/issues/comments/3124002704/reactions
2025-07-27T05:39:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3124002704
https://avatars.githubusercontent.com/u/183703408?v=4
https://api.github.com/users/Sanjaykumar030/events{/privacy}
https://api.github.com/users/Sanjaykumar030/followers
https://api.github.com/users/Sanjaykumar030/following{/other_user}
https://api.github.com/users/Sanjaykumar030/gists{/gist_id}
https://github.com/Sanjaykumar030
Sanjaykumar030
U_kgDOCvMXcA
https://api.github.com/users/Sanjaykumar030/orgs
https://api.github.com/users/Sanjaykumar030/received_events
https://api.github.com/users/Sanjaykumar030/repos
https://api.github.com/users/Sanjaykumar030/starred{/owner}{/repo}
https://api.github.com/users/Sanjaykumar030/subscriptions
User
https://api.github.com/users/Sanjaykumar030
public
## Description

The current documentation example for `datasets.Dataset.map()` demonstrates batched processing but uses a `tokenizer` object without defining or importing it. This causes an error every time it's copied.

Here is the problematic line:

```python
# process a batch of examples
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This assumes the user has already set up a tokenizer, which contradicts the goal of having self-contained, copy-paste-friendly examples. 

## Problem
Users who copy and run the example as-is will encounter:

```python
NameError: name 'tokenizer' is not defined
```

This breaks the flow for users and violates HuggingFace's documentation principle that examples should "work as expected" when copied directly. 

## Proposal
Update the example to include the required tokenizer setup using the Transformers library, like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
ds_tokenized = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This will help new users understand the workflow and apply the method correctly.

## Note
This PR complements ongoing improvements like #7700, which clarifies multiprocessing in .map(). My change focuses on undefined tokenizer — causes NameError




