Dataset.unique(column: str) -> ArrowNotImplementedError
MEMBER
Hi @orkenstein, thanks for reporting.

Please note that for this case, our `datasets` library uses under the hood the Apache Arrow `unique` function: https://arrow.apache.org/docs/python/generated/pyarrow.compute.unique.html#pyarrow.compute.unique

And currently the Apache Arrow `unique` function is only implemented for these input types (see info in their [docs](https://arrow.apache.org/docs/cpp/compute.html#array-wise-vector-functions)): Boolean, Null, Numeric, Temporal, Binary- and String-like.

However, the data types of the `wikiann` dataset are all `list<item: string>` (see its [dataset card](https://huggingface.co/datasets/wikiann#data-fields)), and thus, not yet supported by the Apache Arrow `unique` function.
2022-03-29T08:21:06Z
https://github.com/huggingface/datasets/issues/4047#issuecomment-1081572389
https://api.github.com/repos/huggingface/datasets/issues/4047
IC_kwDODunzps5Ad3wl
https://api.github.com/repos/huggingface/datasets/issues/comments/1081572389/reactions
2022-03-29T08:21:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1081572389
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
As a workaround solution you can use pandas:
```python
from datasets import load_dataset

dataset = load_dataset('wikiann', 'en', split='train')
df = dataset.to_pandas()
unique_df = df[~df.tokens.apply(tuple).duplicated()]  # from https://stackoverflow.com/a/46958336/17517845
```

Note that pandas loads the dataset in memory (this one is small so it's fine).
2022-04-01T16:38:17Z
https://github.com/huggingface/datasets/issues/4047#issuecomment-1086122136
https://api.github.com/repos/huggingface/datasets/issues/4047
IC_kwDODunzps5AvOiY
https://api.github.com/repos/huggingface/datasets/issues/comments/1086122136/reactions
2022-04-01T16:38:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1086122136
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq thank you! I will fall back to this method for now
2022-04-01T18:24:57Z
https://github.com/huggingface/datasets/issues/4047#issuecomment-1086206859
https://api.github.com/repos/huggingface/datasets/issues/4047
IC_kwDODunzps5AvjOL
https://api.github.com/repos/huggingface/datasets/issues/comments/1086206859/reactions
2022-04-01T18:24:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1086206859
https://avatars.githubusercontent.com/u/1461936?v=4
https://api.github.com/users/orkenstein/events{/privacy}
https://api.github.com/users/orkenstein/followers
https://api.github.com/users/orkenstein/following{/other_user}
https://api.github.com/users/orkenstein/gists{/gist_id}
https://github.com/orkenstein
orkenstein
MDQ6VXNlcjE0NjE5MzY=
https://api.github.com/users/orkenstein/orgs
https://api.github.com/users/orkenstein/received_events
https://api.github.com/users/orkenstein/repos
https://api.github.com/users/orkenstein/starred{/owner}{/repo}
https://api.github.com/users/orkenstein/subscriptions
User
https://api.github.com/users/orkenstein
public
## Describe the bug
I'm trying to use `unique()` function, but it fails
## Steps to reproduce the bug
1. Get dataset
2. Call `unique`
3. Error

# Sample code to reproduce the bug
```python
!pip show datasets

from datasets import load_dataset

dataset = load_dataset('wikiann', 'en')

dataset['train'].column_names
dataset['train'].unique(dataset['train'].column_names[0])
```

## Expected results
It would be nice to actually see unique items

## Actual results
Error:
```python
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
[<ipython-input-10-5e0de07ed42c>](https://s0qyv2vjaji-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220324-060046-RC00_436956229#) in <module>()
      6 
      7 dataset['train'].column_names
----> 8 dataset['train'].unique(dataset['train'].column_names[0])

5 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Function unique has no kernel matching input types (array[list<item: string>])
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Google Collab
- Python version: 3.7.13
- PyArrow version: 6.0.1
