Add option to find unique elements in nested sequences when calling `Dataset.unique`
MEMBER
Hi @mariosasko!

Has this been patched into any of the releases?
2022-05-24T02:16:15Z
https://github.com/huggingface/datasets/issues/3299#issuecomment-1135324936
https://api.github.com/repos/huggingface/datasets/issues/3299
IC_kwDODunzps5Dq68I
https://api.github.com/repos/huggingface/datasets/issues/comments/1135324936/reactions
2022-05-24T02:16:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1135324936
https://avatars.githubusercontent.com/u/22957388?v=4
https://api.github.com/users/sayakpaul/events{/privacy}
https://api.github.com/users/sayakpaul/followers
https://api.github.com/users/sayakpaul/following{/other_user}
https://api.github.com/users/sayakpaul/gists{/gist_id}
https://github.com/sayakpaul
sayakpaul
MDQ6VXNlcjIyOTU3Mzg4
https://api.github.com/users/sayakpaul/orgs
https://api.github.com/users/sayakpaul/received_events
https://api.github.com/users/sayakpaul/repos
https://api.github.com/users/sayakpaul/starred{/owner}{/repo}
https://api.github.com/users/sayakpaul/subscriptions
User
https://api.github.com/users/sayakpaul
public
COLLABORATOR
Hi! Not yet, would you be interested in contributing a PR? I can give you some pointers if needed. 
2022-05-24T11:31:47Z
https://github.com/huggingface/datasets/issues/3299#issuecomment-1135796805
https://api.github.com/repos/huggingface/datasets/issues/3299
IC_kwDODunzps5DsuJF
https://api.github.com/repos/huggingface/datasets/issues/comments/1135796805/reactions
2022-05-24T11:31:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1135796805
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
@mariosasko did this ever get implemented? Willing to help if you are still up for it.
2023-05-19T00:03:12Z
https://github.com/huggingface/datasets/issues/3299#issuecomment-1553814258
https://api.github.com/repos/huggingface/datasets/issues/3299
IC_kwDODunzps5cnVLy
https://api.github.com/repos/huggingface/datasets/issues/comments/1553814258/reactions
2023-05-19T00:03:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1553814258
https://avatars.githubusercontent.com/u/6372983?v=4
https://api.github.com/users/ZQ-Dev8/events{/privacy}
https://api.github.com/users/ZQ-Dev8/followers
https://api.github.com/users/ZQ-Dev8/following{/other_user}
https://api.github.com/users/ZQ-Dev8/gists{/gist_id}
https://github.com/ZQ-Dev8
ZQ-Dev8
MDQ6VXNlcjYzNzI5ODM=
https://api.github.com/users/ZQ-Dev8/orgs
https://api.github.com/users/ZQ-Dev8/received_events
https://api.github.com/users/ZQ-Dev8/repos
https://api.github.com/users/ZQ-Dev8/starred{/owner}{/repo}
https://api.github.com/users/ZQ-Dev8/subscriptions
User
https://api.github.com/users/ZQ-Dev8
public
COLLABORATOR
@dcruiz01 No, but here is an example of how to do this with the existing API:


```python
ds = Dataset.from_dict({"tokens": [["a", "b"], ["c", "a"], ["c", "e"]]})

def flatten_tokens(pa_table):
    return pa.table([pc.list_flatten(pa_table["tokens"])], ["flat_tokens"])

ds = ds.with_format("arrow")
ds = ds.map(flatten_tokens, batched=True)
ds = ds.with_format(None)

unique_tokens = ds.unique("flat_tokens")
```

When I think about it, `.unique` on `Sequence(Value(...))` should return unique sequences/arrays, not unique elements of these sequences...
2023-05-19T14:45:39Z
https://github.com/huggingface/datasets/issues/3299#issuecomment-1554701964
https://api.github.com/repos/huggingface/datasets/issues/3299
IC_kwDODunzps5cqt6M
https://api.github.com/repos/huggingface/datasets/issues/comments/1554701964/reactions
2023-05-19T14:45:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1554701964
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
It would be nice to have an option to flatten nested sequences to find unique elements stored in them when calling `Dataset.unique`. ~~Currently, `Dataset.unique` only supports finding unique sequences and not unique elements in that situation.~~