Can datasets remove duplicated rows?
MEMBER
Hi ! For now this is probably the best option.
We might add a feature like this in the feature as well.

Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases
2021-06-18T15:04:01Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-864102851
https://api.github.com/repos/huggingface/datasets/issues/2514
MDEyOklzc3VlQ29tbWVudDg2NDEwMjg1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/864102851/reactions
2021-06-18T15:04:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864102851
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Yes, I'd like to work on this feature once I'm done with #2500, but first I have to do some research, and see if the implementation wouldn't be too complex.

In the meantime, maybe [this lib](https://github.com/TomScheffers/pyarrow_ops) can help. However, note that this lib operates directly on pyarrow tables and relies only on `hash` to find duplicates (e.g. `-1` and `-2` have the same hash in Python 3, so this lib will treat them as duplicates), which doesn't make much sense.
2021-06-19T00:01:57Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-864324610
https://api.github.com/repos/huggingface/datasets/issues/2514
MDEyOklzc3VlQ29tbWVudDg2NDMyNDYxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/864324610/reactions
2021-06-19T00:01:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864324610
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> Hi ! For now this is probably the best option.
> We might add a feature like this in the feature as well.
> 
> Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
> Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases

Great if this is can be done. Thanks!!

Not sure if you are asking me. In any case I don't know of any unfortunately :( in practice if data is really large we normally do it with spark (only for info. I understand this is not useful in developing this library..)
2021-06-21T07:37:04Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-864802596
https://api.github.com/repos/huggingface/datasets/issues/2514
MDEyOklzc3VlQ29tbWVudDg2NDgwMjU5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/864802596/reactions
2021-06-21T07:37:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864802596
https://avatars.githubusercontent.com/u/16516583?v=4
https://api.github.com/users/liuxinglan/events{/privacy}
https://api.github.com/users/liuxinglan/followers
https://api.github.com/users/liuxinglan/following{/other_user}
https://api.github.com/users/liuxinglan/gists{/gist_id}
https://github.com/liuxinglan
liuxinglan
MDQ6VXNlcjE2NTE2NTgz
https://api.github.com/users/liuxinglan/orgs
https://api.github.com/users/liuxinglan/received_events
https://api.github.com/users/liuxinglan/repos
https://api.github.com/users/liuxinglan/starred{/owner}{/repo}
https://api.github.com/users/liuxinglan/subscriptions
User
https://api.github.com/users/liuxinglan
public
CONTRIBUTOR
Hello,

I'm also interested in this feature.
Has there been progress on this issue?

Could we use a similar trick as above, but with a better hashing algorithm like SHA?

We could also use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), should we care a lot about collision in this case?
2021-10-07T20:42:39Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-938141146
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps436uXa
https://api.github.com/repos/huggingface/datasets/issues/comments/938141146/reactions
2021-10-11T16:13:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/938141146
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
CONTRIBUTOR
For reference, we can get a solution fairly easily if we assume that we can hold in memory all unique values. 

```python
from datasets import Dataset
from itertools import cycle
from functools import partial

memory = set()
def is_unique(elem:Any , column: str, memory: set) -> bool:
    if elem[column] in memory:
        return False
    else:
        memory.add(elem[column])
        return True

# Example dataset
ds = Dataset.from_dict({"col1" : [sent for i, sent in zip(range(10), cycle(["apple", "orange", "pear"]))],
                                      "col2": [i % 5 for i in range(10)]})

# Drop duplicates in `ds` on "col1"
ds2 = ds.filter(partial(is_unique, column="col1", memory=memory))
```

Of course, we can improve the API so that we can introduce `Dataset.drop_duplicates`.
For the parallel version, we can use a shared memory set.
2021-11-06T19:08:26Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-962496585
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps45XohJ
https://api.github.com/repos/huggingface/datasets/issues/comments/962496585/reactions
2021-11-06T19:08:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/962496585
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
MEMBER
An approach that works assuming you can hold the all the unique document hashes in memory:
```python
from datasets import load_dataset

def get_hash(example):
    """Get hash of content field."""
    return {"hash": hash(example["content"])} # can use any hashing function here
    
def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:
        return False

ds = load_dataset("some_dataset")
ds = ds.map(get_hash)
uniques = set(ds.unique("hash"))
ds_filter = ds.filter(check_uniques, fn_kwargs={"uniques": uniques})
```
If the `uniques` could be stored in arrow then no additional memory would used at all but I don't know if this is possible.

2021-12-02T08:39:21Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-984409349
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps46rOUF
https://api.github.com/repos/huggingface/datasets/issues/comments/984409349/reactions
2021-12-02T08:39:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/984409349
https://avatars.githubusercontent.com/u/8264887?v=4
https://api.github.com/users/lvwerra/events{/privacy}
https://api.github.com/users/lvwerra/followers
https://api.github.com/users/lvwerra/following{/other_user}
https://api.github.com/users/lvwerra/gists{/gist_id}
https://github.com/lvwerra
lvwerra
MDQ6VXNlcjgyNjQ4ODc=
https://api.github.com/users/lvwerra/orgs
https://api.github.com/users/lvwerra/received_events
https://api.github.com/users/lvwerra/repos
https://api.github.com/users/lvwerra/starred{/owner}{/repo}
https://api.github.com/users/lvwerra/subscriptions
User
https://api.github.com/users/lvwerra
public
NONE
@lvwerra hey, could you tell me how reliable is this deduplication method. i am currently using the same deduplication strategy to deduplicate a large text corpus to pretrain LLMs ~ 11B to 20B. just needed to ensure if this strategy would be fine on large datasets for LLMs pretraining. 
2022-09-10T14:43:26Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-1242744669
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps5KEsdd
https://api.github.com/repos/huggingface/datasets/issues/comments/1242744669/reactions
2022-09-10T14:43:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1242744669
https://avatars.githubusercontent.com/u/32235549?v=4
https://api.github.com/users/StephennFernandes/events{/privacy}
https://api.github.com/users/StephennFernandes/followers
https://api.github.com/users/StephennFernandes/following{/other_user}
https://api.github.com/users/StephennFernandes/gists{/gist_id}
https://github.com/StephennFernandes
StephennFernandes
MDQ6VXNlcjMyMjM1NTQ5
https://api.github.com/users/StephennFernandes/orgs
https://api.github.com/users/StephennFernandes/received_events
https://api.github.com/users/StephennFernandes/repos
https://api.github.com/users/StephennFernandes/starred{/owner}{/repo}
https://api.github.com/users/StephennFernandes/subscriptions
User
https://api.github.com/users/StephennFernandes
public
NONE
Hi @StephennFernandes  I'm also trying to pretrain an llm, and need to do deduplication for my dataset, 
which method you applied please?
2024-01-25T13:46:52Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-1910253181
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps5x3CZ9
https://api.github.com/repos/huggingface/datasets/issues/comments/1910253181/reactions
2024-01-25T13:46:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1910253181
https://avatars.githubusercontent.com/u/47546147?v=4
https://api.github.com/users/Manel-Hik/events{/privacy}
https://api.github.com/users/Manel-Hik/followers
https://api.github.com/users/Manel-Hik/following{/other_user}
https://api.github.com/users/Manel-Hik/gists{/gist_id}
https://github.com/Manel-Hik
Manel-Hik
MDQ6VXNlcjQ3NTQ2MTQ3
https://api.github.com/users/Manel-Hik/orgs
https://api.github.com/users/Manel-Hik/received_events
https://api.github.com/users/Manel-Hik/repos
https://api.github.com/users/Manel-Hik/starred{/owner}{/repo}
https://api.github.com/users/Manel-Hik/subscriptions
User
https://api.github.com/users/Manel-Hik
public
NONE
Hey @Manel-Hik 

The following is a simpler yet really effective deduplication code that i has used in the past. 

given that I have limited training corpus for the languages I wanted to train i relied on this code. https://huggingface.co/datasets/Finnish-NLP/mc4_fi_cleaned/blob/main/deduplicate.py


for more robust and stronger deduplication, refer to this huggingface repo, that's newly released: https://github.com/huggingface/datatrove




2024-01-25T14:20:53Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-1910310844
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps5x3Qe8
https://api.github.com/repos/huggingface/datasets/issues/comments/1910310844/reactions
2024-01-25T14:24:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1910310844
https://avatars.githubusercontent.com/u/32235549?v=4
https://api.github.com/users/StephennFernandes/events{/privacy}
https://api.github.com/users/StephennFernandes/followers
https://api.github.com/users/StephennFernandes/following{/other_user}
https://api.github.com/users/StephennFernandes/gists{/gist_id}
https://github.com/StephennFernandes
StephennFernandes
MDQ6VXNlcjMyMjM1NTQ5
https://api.github.com/users/StephennFernandes/orgs
https://api.github.com/users/StephennFernandes/received_events
https://api.github.com/users/StephennFernandes/repos
https://api.github.com/users/StephennFernandes/starred{/owner}{/repo}
https://api.github.com/users/StephennFernandes/subscriptions
User
https://api.github.com/users/StephennFernandes
public
NONE
Thanks a lot Sure I will check it @StephennFernandes 
2024-01-25T14:27:58Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-1910323492
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps5x3Tkk
https://api.github.com/repos/huggingface/datasets/issues/comments/1910323492/reactions
2024-01-25T14:27:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1910323492
https://avatars.githubusercontent.com/u/47546147?v=4
https://api.github.com/users/Manel-Hik/events{/privacy}
https://api.github.com/users/Manel-Hik/followers
https://api.github.com/users/Manel-Hik/following{/other_user}
https://api.github.com/users/Manel-Hik/gists{/gist_id}
https://github.com/Manel-Hik
Manel-Hik
MDQ6VXNlcjQ3NTQ2MTQ3
https://api.github.com/users/Manel-Hik/orgs
https://api.github.com/users/Manel-Hik/received_events
https://api.github.com/users/Manel-Hik/repos
https://api.github.com/users/Manel-Hik/starred{/owner}{/repo}
https://api.github.com/users/Manel-Hik/subscriptions
User
https://api.github.com/users/Manel-Hik
public
CONTRIBUTOR
Hi, is there any updates? Thanks!
2024-03-28T05:37:40Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-2024444283
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps54qpF7
https://api.github.com/repos/huggingface/datasets/issues/comments/2024444283/reactions
2024-03-28T05:37:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2024444283
https://avatars.githubusercontent.com/u/5236035?v=4
https://api.github.com/users/fzyzcjy/events{/privacy}
https://api.github.com/users/fzyzcjy/followers
https://api.github.com/users/fzyzcjy/following{/other_user}
https://api.github.com/users/fzyzcjy/gists{/gist_id}
https://github.com/fzyzcjy
fzyzcjy
MDQ6VXNlcjUyMzYwMzU=
https://api.github.com/users/fzyzcjy/orgs
https://api.github.com/users/fzyzcjy/received_events
https://api.github.com/users/fzyzcjy/repos
https://api.github.com/users/fzyzcjy/starred{/owner}{/repo}
https://api.github.com/users/fzyzcjy/subscriptions
User
https://api.github.com/users/fzyzcjy
public
CONTRIBUTOR
Update July 2024

PyArrow now supports `first`/`last` aggregation which would allow us to implement this functionality. [Link](https://github.com/apache/arrow/issues/30950)

so if we want to move in this direction we can :) Is that something we want to do? Would be happy to contribute.
2024-07-19T13:23:00Z
https://github.com/huggingface/datasets/issues/2514#issuecomment-2239170584
https://api.github.com/repos/huggingface/datasets/issues/2514
IC_kwDODunzps6FdwgY
https://api.github.com/repos/huggingface/datasets/issues/comments/2239170584/reactions
2024-07-19T13:23:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2239170584
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..


**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...


**Additional context**
no