to_tf_dataset with a TF collator causes bizarrely persistent slowdown
MEMBER
wtf
2023-01-23T10:21:41Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1400110404
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5Tc_1E
https://api.github.com/repos/huggingface/datasets/issues/comments/1400110404/reactions
2023-01-23T10:21:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1400110404
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Couldn't find what's causing this, this will need more investigation
2023-01-23T11:13:00Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1400174957
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5TdPlt
https://api.github.com/repos/huggingface/datasets/issues/comments/1400174957/reactions
2023-01-23T11:13:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1400174957
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
A possible hint: The function it seems to be spending a lot of time in (when iterating over the original dataset) is `_get_mp` in the PIL JPEG decoder: 
![image](https://user-images.githubusercontent.com/12866554/214057267-c889f05e-efaf-4036-b805-c5381fa62f4a.png)

2023-01-23T13:55:58Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1400391303
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5TeEaH
https://api.github.com/repos/huggingface/datasets/issues/comments/1400391303/reactions
2023-01-23T13:55:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1400391303
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
If "mp" is multiprocessing, this might suggest some kind of negative interaction between the JPEG decoder and TF's handling of processes/threads. Note that we haven't merged the parallel `to_tf_dataset` PR yet, so it's not caused by that PR!
2023-01-23T13:57:22Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1400393323
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5TeE5r
https://api.github.com/repos/huggingface/datasets/issues/comments/1400393323/reactions
2023-01-23T13:57:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1400393323
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Update: MP isn't multiprocessing at all, it's an internal PIL method for loading metadata from JPEG files. No idea why that would be a bottleneck, but I'll see if a Python profiler can't figure out where the time is actually being spent.
2023-01-23T17:36:10Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1400726801
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5TfWUR
https://api.github.com/repos/huggingface/datasets/issues/comments/1400726801/reactions
2023-01-23T17:36:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1400726801
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
After further profiling, the slowdown is in the C methods for JPEG decoding that are included as part of PIL. Because Python profilers can't inspect inside that, I don't have any further information on which lines exactly are responsible for the slowdown or why.

In the meantime, I'm going to suggest switching from `return_tensors="tf"` to `return_tensors="np"` in most of our `transformers` code - this generally works better for pre-processing. Two relevant PRs are [here](https://github.com/huggingface/transformers/pull/21266) and [here](https://github.com/huggingface/notebooks/pull/308).
2023-01-23T18:54:09Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1400822052
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5Tftkk
https://api.github.com/repos/huggingface/datasets/issues/comments/1400822052/reactions
2023-01-23T18:54:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1400822052
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Closing this issue as we've done what we can with this one! 
2023-02-13T14:13:33Z
https://github.com/huggingface/datasets/issues/5450#issuecomment-1428008074
https://api.github.com/repos/huggingface/datasets/issues/5450
IC_kwDODunzps5VHayK
https://api.github.com/repos/huggingface/datasets/issues/comments/1428008074/reactions
2023-02-13T14:13:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1428008074
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
### Describe the bug

This will make more sense if you take a look at [a Colab notebook that reproduces this issue.](https://colab.research.google.com/drive/1rxyeciQFWJTI0WrZ5aojp4Ls1ut18fNH?usp=sharing) 

Briefly, there are several datasets that, when you iterate over them with `to_tf_dataset` **and** a data collator that returns `tf` tensors, become very slow. We haven't been able to figure this one out - it can be intermittent, and we have no idea what could possibly cause it. The weirdest thing is that **the slowdown affects other attempts to access the underlying dataset**. If you try to iterate over the `tf.data.Dataset`, then interrupt execution, and then try to iterate over the original dataset, the original dataset is now also very slow! This is true even if the dataset format is not set to `tf` - the iteration is slow even though it's not calling TF at all!

There is a simple workaround for this - we can simply get our data collators to return `np` tensors. When we do this, the bug is never triggered and everything is fine. In general, `np` is preferred for this kind of preprocessing work anyway, when the preprocessing is not going to be compiled into a pure `tf.data` pipeline! However, the issue is fascinating, and the TF team were wondering if anyone in datasets (cc @lhoestq @mariosasko) might have an idea of what could cause this.

### Steps to reproduce the bug

Run the attached Colab.

### Expected behavior

The slowdown should go away, or at least not persist after we stop iterating over the `tf.data.Dataset`

### Environment info

The issue occurs on multiple versions of Python and TF, both on local machines and on Colab.

All testing was done using the latest versions of `transformers` and `datasets` from `main`