Avoid PyArrow type optimization if it fails
MEMBER
That's good to have a way to disable this easily :)
I just find it a bit unfortunate that users would have to experience the error once and then do `DISABLE_PYARROW_TYPES_OPTIMIZATION=1`. Do you know if there's a way to simply fallback on disabling it automatically when it fails ?
2021-11-08T16:14:19Z
https://github.com/huggingface/datasets/pull/3234#issuecomment-963315492
https://api.github.com/repos/huggingface/datasets/issues/3234
IC_kwDODunzps45awck
https://api.github.com/repos/huggingface/datasets/issues/comments/963315492/reactions
2021-11-08T16:14:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/963315492
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
@lhoestq Actually, I agree a fallback makes more sense. The current approach is not very practical indeed and would require a mention in the docs.

2021-11-08T16:39:06Z
https://github.com/huggingface/datasets/pull/3234#issuecomment-963347953
https://api.github.com/repos/huggingface/datasets/issues/3234
IC_kwDODunzps45a4Xx
https://api.github.com/repos/huggingface/datasets/issues/comments/963347953/reactions
2021-11-08T16:39:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/963347953
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
COLLABORATOR
Replaced the env variable with a fallback!
2021-11-08T18:07:20Z
https://github.com/huggingface/datasets/pull/3234#issuecomment-963424883
https://api.github.com/repos/huggingface/datasets/issues/3234
IC_kwDODunzps45bLJz
https://api.github.com/repos/huggingface/datasets/issues/comments/963424883/reactions
2021-11-08T21:51:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/963424883
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Hmm if the fallback automatically happens without the user knowing it, then I don't think we really need to mention it. But if you really wanted to, I think the [Improve performance](https://huggingface.co/docs/datasets/cache.html#improve-performance) section would be a great place for it! 
2021-11-09T19:43:01Z
https://github.com/huggingface/datasets/pull/3234#issuecomment-964477118
https://api.github.com/repos/huggingface/datasets/issues/3234
IC_kwDODunzps45fMC-
https://api.github.com/repos/huggingface/datasets/issues/comments/964477118/reactions
2021-11-09T19:43:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/964477118
https://avatars.githubusercontent.com/u/59462357?v=4
https://api.github.com/users/stevhliu/events{/privacy}
https://api.github.com/users/stevhliu/followers
https://api.github.com/users/stevhliu/following{/other_user}
https://api.github.com/users/stevhliu/gists{/gist_id}
https://github.com/stevhliu
stevhliu
MDQ6VXNlcjU5NDYyMzU3
https://api.github.com/users/stevhliu/orgs
https://api.github.com/users/stevhliu/received_events
https://api.github.com/users/stevhliu/repos
https://api.github.com/users/stevhliu/starred{/owner}{/repo}
https://api.github.com/users/stevhliu/subscriptions
User
https://api.github.com/users/stevhliu
public
MEMBER
Yea I think this could just end up in a note that says that `datasets` automatically picks the most optimized integer precision for your tokenized text data to save you disk space. Maybe later if we have a page on text processing we can add this note, but for now I agree it doesn't fit well into the doc.

In particular in the "Improve performance" section we mention what users can do to speed up their computations, while this behavior is just some internal feature that users don't have control over anyway.
2021-11-10T12:03:48Z
https://github.com/huggingface/datasets/pull/3234#issuecomment-965066022
https://api.github.com/repos/huggingface/datasets/issues/3234
IC_kwDODunzps45hb0m
https://api.github.com/repos/huggingface/datasets/issues/comments/965066022/reactions
2021-11-10T12:03:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/965066022
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Adds a new variable, `DISABLE_PYARROW_TYPES_OPTIMIZATION`, to `config.py` for easier control of the Arrow type optimization.

Fix #2206  