Bugs : dataset.map() is frozen on ELI5
MEMBER
This comes from an overflow in pyarrow's array.
It is stuck inside the loop that reduces the batch size to avoid the overflow.
I'll take a look
2020-08-11T15:31:43Z
https://github.com/huggingface/datasets/issues/482#issuecomment-672019148
https://api.github.com/repos/huggingface/datasets/issues/482
MDEyOklzc3VlQ29tbWVudDY3MjAxOTE0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/672019148/reactions
2020-08-11T15:31:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672019148
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I created a PR to fix the issue.
It was due to an overflow check that handled badly an empty list.

You can try the changes by using 
```
!pip install git+https://github.com/huggingface/nlp.git@fix-bad-type-in-overflow-check
```

Also I noticed that the first 1000 examples have an empty list in the `title_urls` field. The feature type inference in `.map` will consider it `null` because of that, and it will crash when it encounter the next example with a `title_urls` that is not empty.

Therefore to fix that, what you can do for now is increase the writer batch size so that the feature inference will take into account at least one example with a non-empty `title_urls`:

```python
# default batch size is 1_000 and it's not enough for feature type inference because of empty lists
valid_dataset = valid_dataset.map(make_input_target, writer_batch_size=3_000) 
```

I was able to run the frozen cell with these changes.
2020-08-11T16:34:32Z
https://github.com/huggingface/datasets/issues/482#issuecomment-672074634
https://api.github.com/repos/huggingface/datasets/issues/482
MDEyOklzc3VlQ29tbWVudDY3MjA3NDYzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/672074634/reactions
2020-08-11T16:34:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672074634
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq Perfect and thank you very much!!
Close the issue.
2020-08-11T23:55:11Z
https://github.com/huggingface/datasets/issues/482#issuecomment-672372670
https://api.github.com/repos/huggingface/datasets/issues/482
MDEyOklzc3VlQ29tbWVudDY3MjM3MjY3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/672372670/reactions
2020-08-11T23:55:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672372670
https://avatars.githubusercontent.com/u/56621342?v=4
https://api.github.com/users/ratthachat/events{/privacy}
https://api.github.com/users/ratthachat/followers
https://api.github.com/users/ratthachat/following{/other_user}
https://api.github.com/users/ratthachat/gists{/gist_id}
https://github.com/ratthachat
ratthachat
MDQ6VXNlcjU2NjIxMzQy
https://api.github.com/users/ratthachat/orgs
https://api.github.com/users/ratthachat/received_events
https://api.github.com/users/ratthachat/repos
https://api.github.com/users/ratthachat/starred{/owner}{/repo}
https://api.github.com/users/ratthachat/subscriptions
User
https://api.github.com/users/ratthachat
public
NONE
@lhoestq mapping the function `make_input_target` was passed by your fixing.

However, there is another error in the final step of `valid_dataset.map(convert_to_features, batched=True)`

`ArrowInvalid: Could not convert Thepiratebay.vg with type str: converting to null type`
(The [same colab notebook above with new error message](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing#scrollTo=5sRrJ3_C8rLt))

Do you have some ideas? (I am really sorry I could not debug it by myself since I never used `pyarrow` before) 
Note that `train_dataset.map(convert_to_features, batched=True)` can be run successfully even though train_dataset is 27x bigger than `valid_dataset` so I believe the problem lies in some field of `valid_dataset` again .
2020-08-12T00:08:28Z
https://github.com/huggingface/datasets/issues/482#issuecomment-672380921
https://api.github.com/repos/huggingface/datasets/issues/482
MDEyOklzc3VlQ29tbWVudDY3MjM4MDkyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/672380921/reactions
2020-08-12T00:09:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672380921
https://avatars.githubusercontent.com/u/56621342?v=4
https://api.github.com/users/ratthachat/events{/privacy}
https://api.github.com/users/ratthachat/followers
https://api.github.com/users/ratthachat/following{/other_user}
https://api.github.com/users/ratthachat/gists{/gist_id}
https://github.com/ratthachat
ratthachat
MDQ6VXNlcjU2NjIxMzQy
https://api.github.com/users/ratthachat/orgs
https://api.github.com/users/ratthachat/received_events
https://api.github.com/users/ratthachat/repos
https://api.github.com/users/ratthachat/starred{/owner}{/repo}
https://api.github.com/users/ratthachat/subscriptions
User
https://api.github.com/users/ratthachat
public
MEMBER
I got this issue too and fixed it by specifying `writer_batch_size=3_000` in `.map`.
This is because Arrow didn't expect `Thepiratebay.vg` in `title_urls `, as all previous examples have empty lists in `title_urls `
2020-08-12T10:13:43Z
https://github.com/huggingface/datasets/issues/482#issuecomment-672784061
https://api.github.com/repos/huggingface/datasets/issues/482
MDEyOklzc3VlQ29tbWVudDY3Mjc4NDA2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/672784061/reactions
2020-08-12T10:13:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672784061
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I am clear now . Thank so much again Quentin!
2020-08-12T14:13:46Z
https://github.com/huggingface/datasets/issues/482#issuecomment-672896694
https://api.github.com/repos/huggingface/datasets/issues/482
MDEyOklzc3VlQ29tbWVudDY3Mjg5NjY5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/672896694/reactions
2020-08-12T14:13:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672896694
https://avatars.githubusercontent.com/u/56621342?v=4
https://api.github.com/users/ratthachat/events{/privacy}
https://api.github.com/users/ratthachat/followers
https://api.github.com/users/ratthachat/following{/other_user}
https://api.github.com/users/ratthachat/gists{/gist_id}
https://github.com/ratthachat
ratthachat
MDQ6VXNlcjU2NjIxMzQy
https://api.github.com/users/ratthachat/orgs
https://api.github.com/users/ratthachat/received_events
https://api.github.com/users/ratthachat/repos
https://api.github.com/users/ratthachat/starred{/owner}{/repo}
https://api.github.com/users/ratthachat/subscriptions
User
https://api.github.com/users/ratthachat
public
NONE
I'm getting a hanging `dataset.map()` when running a gradio app with `gradio` for auto-reloading instead of `python`
2023-04-05T20:26:34Z
https://github.com/huggingface/datasets/issues/482#issuecomment-1498103514
https://api.github.com/repos/huggingface/datasets/issues/482
IC_kwDODunzps5ZSz7a
https://api.github.com/repos/huggingface/datasets/issues/comments/1498103514/reactions
2023-04-05T20:26:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1498103514
https://avatars.githubusercontent.com/u/1486609?v=4
https://api.github.com/users/lxe/events{/privacy}
https://api.github.com/users/lxe/followers
https://api.github.com/users/lxe/following{/other_user}
https://api.github.com/users/lxe/gists{/gist_id}
https://github.com/lxe
lxe
MDQ6VXNlcjE0ODY2MDk=
https://api.github.com/users/lxe/orgs
https://api.github.com/users/lxe/received_events
https://api.github.com/users/lxe/repos
https://api.github.com/users/lxe/starred{/owner}{/repo}
https://api.github.com/users/lxe/subscriptions
User
https://api.github.com/users/lxe
public
MEMBER
Maybe this is an issue with gradio, could you open an issue on their repo ? `Dataset.map` simply uses `multiprocess.Pool` for multiprocessing

If you interrupt the program mayeb the stack trace would give some information of where it was hanging in the code (maybe a lock somewhere ?)
2023-04-06T09:39:59Z
https://github.com/huggingface/datasets/issues/482#issuecomment-1498774265
https://api.github.com/repos/huggingface/datasets/issues/482
IC_kwDODunzps5ZVXr5
https://api.github.com/repos/huggingface/datasets/issues/comments/1498774265/reactions
2023-04-06T09:39:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1498774265
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) 

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?