Request for text deduplication feature
COLLABORATOR
The "exact match" deduplication will be possible when we resolve https://github.com/huggingface/datasets/issues/2514 (first, https://github.com/apache/arrow/issues/30950 needs to be addressed on the Arrow side). In the meantime, you can use Polars or DuckDB (e.g., via [datasets-sql](https://github.com/mariosasko/datasets_sql)).

Fuzzy deduplication is out-of-scope for now ([splink](https://github.com/moj-analytical-services/splink) is probably the best tool for it).
2023-05-31T14:21:18Z
https://github.com/huggingface/datasets/issues/5877#issuecomment-1570335978
https://api.github.com/repos/huggingface/datasets/issues/5877
IC_kwDODunzps5dmWzq
https://api.github.com/repos/huggingface/datasets/issues/comments/1570335978/reactions
2023-05-31T14:21:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1570335978
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
This library can be an intermediate solution : https://github.com/ChenghaoMou/text-dedup/tree/main
2023-06-01T20:26:18Z
https://github.com/huggingface/datasets/issues/5877#issuecomment-1572731181
https://api.github.com/repos/huggingface/datasets/issues/5877
IC_kwDODunzps5dvfkt
https://api.github.com/repos/huggingface/datasets/issues/comments/1572731181/reactions
2023-06-01T20:26:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1572731181
https://avatars.githubusercontent.com/u/1519077?v=4
https://api.github.com/users/MaveriQ/events{/privacy}
https://api.github.com/users/MaveriQ/followers
https://api.github.com/users/MaveriQ/following{/other_user}
https://api.github.com/users/MaveriQ/gists{/gist_id}
https://github.com/MaveriQ
MaveriQ
MDQ6VXNlcjE1MTkwNzc=
https://api.github.com/users/MaveriQ/orgs
https://api.github.com/users/MaveriQ/received_events
https://api.github.com/users/MaveriQ/repos
https://api.github.com/users/MaveriQ/starred{/owner}{/repo}
https://api.github.com/users/MaveriQ/subscriptions
User
https://api.github.com/users/MaveriQ
public
NONE
I have been using polars to remove duplicates but it would be nice to do it directly in pyarrow.

For example,

1. Read dataset with pyarrow
2. Use scan_pyarrow_dataset() with Polars to create a LazyFrame
3. Use sort and unique to remove duplicates based on a subset of columns
4. Convert to table and save data with ds.write_dataset()

There are times where that workflow makes perfect sense because I do additional transformations with Polars. Most of the time I am simply just reading dataset A and writing dataset B without duplicates though, and I wish I could use a pyarrow scanner or table directly. 
2023-07-26T21:42:14Z
https://github.com/huggingface/datasets/issues/5877#issuecomment-1652536254
https://api.github.com/repos/huggingface/datasets/issues/5877
IC_kwDODunzps5if7O-
https://api.github.com/repos/huggingface/datasets/issues/comments/1652536254/reactions
2023-07-26T21:42:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1652536254
https://avatars.githubusercontent.com/u/12517855?v=4
https://api.github.com/users/ldacey/events{/privacy}
https://api.github.com/users/ldacey/followers
https://api.github.com/users/ldacey/following{/other_user}
https://api.github.com/users/ldacey/gists{/gist_id}
https://github.com/ldacey
ldacey
MDQ6VXNlcjEyNTE3ODU1
https://api.github.com/users/ldacey/orgs
https://api.github.com/users/ldacey/received_events
https://api.github.com/users/ldacey/repos
https://api.github.com/users/ldacey/starred{/owner}{/repo}
https://api.github.com/users/ldacey/subscriptions
User
https://api.github.com/users/ldacey
public
NONE
Hi
see this new release  from hf [datatrove](https://github.com/huggingface/datatrove)
DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality
2024-01-25T14:39:33Z
https://github.com/huggingface/datasets/issues/5877#issuecomment-1910343859
https://api.github.com/repos/huggingface/datasets/issues/5877
IC_kwDODunzps5x3Yiz
https://api.github.com/repos/huggingface/datasets/issues/comments/1910343859/reactions
2024-01-25T14:40:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1910343859
https://avatars.githubusercontent.com/u/47546147?v=4
https://api.github.com/users/Manel-Hik/events{/privacy}
https://api.github.com/users/Manel-Hik/followers
https://api.github.com/users/Manel-Hik/following{/other_user}
https://api.github.com/users/Manel-Hik/gists{/gist_id}
https://github.com/Manel-Hik
Manel-Hik
MDQ6VXNlcjQ3NTQ2MTQ3
https://api.github.com/users/Manel-Hik/orgs
https://api.github.com/users/Manel-Hik/received_events
https://api.github.com/users/Manel-Hik/repos
https://api.github.com/users/Manel-Hik/starred{/owner}{/repo}
https://api.github.com/users/Manel-Hik/subscriptions
User
https://api.github.com/users/Manel-Hik
public
### Feature request

It would be great if there would be support for high performance, highly scalable text deduplication algorithms as part of the datasets library.

### Motivation

Motivated by this blog post https://huggingface.co/blog/dedup and this library https://github.com/google-research/deduplicate-text-datasets, but slightly frustrated by how its not very easy to work with these tools I am proposing this feature.

### Your contribution

I would be happy to contribute to the development effort of this feature. would love to collaborate with others in the development effort. 