Issue with pyarrow 14.0.1
MEMBER
Looks like we should stop using `PyExtensionType` and use `ExtensionType` instead

see https://github.com/apache/arrow/commit/f14170976372436ec1d03a724d8d3f3925484ecf
2023-11-10T11:05:56Z
https://github.com/huggingface/datasets/issues/6396#issuecomment-1805522636
https://api.github.com/repos/huggingface/datasets/issues/6396
IC_kwDODunzps5rnhbM
https://api.github.com/repos/huggingface/datasets/issues/comments/1805522636/reactions
2023-11-10T11:07:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1805522636
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
https://github.com/huggingface/datasets-server/pull/2089#pullrequestreview-1724449532

> Yes, I understand now: they have disabled their `PyExtensionType` and we use it in `datasets` for arrays... 
2023-11-10T13:22:15Z
https://github.com/huggingface/datasets/issues/6396#issuecomment-1805718285
https://api.github.com/repos/huggingface/datasets/issues/6396
IC_kwDODunzps5roRMN
https://api.github.com/repos/huggingface/datasets/issues/comments/1805718285/reactions
2023-11-10T13:22:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1805718285
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
related?

https://huggingface.co/datasets/ssbuild/tools_data/discussions/1#654e663b77c8ec680d10479c
2023-11-10T17:20:08Z
https://github.com/huggingface/datasets/issues/6396#issuecomment-1806127939
https://api.github.com/repos/huggingface/datasets/issues/6396
IC_kwDODunzps5rp1ND
https://api.github.com/repos/huggingface/datasets/issues/comments/1806127939/reactions
2023-11-10T17:20:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1806127939
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
> related?
>
> https://huggingface.co/datasets/ssbuild/tools_data/discussions/1#654e663b77c8ec680d10479c

No, related to https://github.com/huggingface/datasets/issues/5706
2023-11-10T18:19:10Z
https://github.com/huggingface/datasets/issues/6396#issuecomment-1806202921
https://api.github.com/repos/huggingface/datasets/issues/6396
IC_kwDODunzps5rqHgp
https://api.github.com/repos/huggingface/datasets/issues/comments/1806202921/reactions
2023-11-10T18:19:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1806202921
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Running the following is a workaround:

```
import pyarrow
pyarrow.PyExtensionType.set_auto_load(True)
```
2023-11-12T00:22:32Z
https://github.com/huggingface/datasets/issues/6396#issuecomment-1806955672
https://api.github.com/repos/huggingface/datasets/issues/6396
IC_kwDODunzps5rs_SY
https://api.github.com/repos/huggingface/datasets/issues/comments/1806955672/reactions
2023-11-12T00:22:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1806955672
https://avatars.githubusercontent.com/u/50503513?v=4
https://api.github.com/users/simonlevine/events{/privacy}
https://api.github.com/users/simonlevine/followers
https://api.github.com/users/simonlevine/following{/other_user}
https://api.github.com/users/simonlevine/gists{/gist_id}
https://github.com/simonlevine
simonlevine
MDQ6VXNlcjUwNTAzNTEz
https://api.github.com/users/simonlevine/orgs
https://api.github.com/users/simonlevine/received_events
https://api.github.com/users/simonlevine/repos
https://api.github.com/users/simonlevine/starred{/owner}{/repo}
https://api.github.com/users/simonlevine/subscriptions
User
https://api.github.com/users/simonlevine
public
See https://github.com/huggingface/datasets-server/pull/2089 for reference

```
from datasets import (Array2D, Dataset, Features)
feature_type = Array2D(shape=(2, 2), dtype="float32")
content = [[0.0, 0.0], [0.0, 0.0]]
features = Features({"col": feature_type})
dataset = Dataset.from_dict({"col": [content]}, features=features)
```

generates

```
/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py:648: FutureWarning: pyarrow.PyExtensionType is deprecated and will refuse deserialization by default. Instead, please derive from pyarrow.ExtensionType and implement your own serialization mechanism.
  pa.PyExtensionType.__init__(self, self.storage_dtype)
/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py:1661: RuntimeWarning: pickle-based deserialization of pyarrow.PyExtensionType subclasses is disabled by default; if you only ingest trusted data files, you may re-enable this using `pyarrow.PyExtensionType.set_auto_load(True)`.
In the future, Python-defined extension subclasses should derive from pyarrow.ExtensionType (not pyarrow.PyExtensionType) and implement their own serialization mechanism.

  obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py:1661: FutureWarning: pyarrow.PyExtensionType is deprecated and will refuse deserialization by default. Instead, please derive from pyarrow.ExtensionType and implement your own serialization mechanism.
  obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 924, in from_dict
    return cls(pa_table, info=info, split=split)
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 693, in __init__
    inferred_features = Features.from_arrow_schema(arrow_table.schema)
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1661, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1661, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1381, in generate_from_arrow_type
    return Value(dtype=_arrow_to_datasets_dtype(pa_type))
  File "/home/slesage/hf/datasets-server/libs/libcommon/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 111, in _arrow_to_datasets_dtype
    raise ValueError(f"Arrow type {arrow_type} does not have a datasets dtype equivalent.")
ValueError: Arrow type extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>> does not have a datasets dtype equivalent.
```