pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
MEMBER
Thanks for reporting @conceptofmind.

Could you please give details about your environment? 
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```
2022-08-03T09:41:48Z
https://github.com/huggingface/datasets/issues/4782#issuecomment-1203721987
https://api.github.com/repos/huggingface/datasets/issues/4782
IC_kwDODunzps5Hv1cD
https://api.github.com/repos/huggingface/datasets/issues/comments/1203721987/reactions
2022-08-03T09:41:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1203721987
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Hi @albertvillanova ,

Here is the environment information:
```
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```
Thanks,

Enrico
2022-08-03T16:44:15Z
https://github.com/huggingface/datasets/issues/4782#issuecomment-1204214482
https://api.github.com/repos/huggingface/datasets/issues/4782
IC_kwDODunzps5HxtrS
https://api.github.com/repos/huggingface/datasets/issues/comments/1204214482/reactions
2022-08-03T16:44:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1204214482
https://avatars.githubusercontent.com/u/25208228?v=4
https://api.github.com/users/conceptofmind/events{/privacy}
https://api.github.com/users/conceptofmind/followers
https://api.github.com/users/conceptofmind/following{/other_user}
https://api.github.com/users/conceptofmind/gists{/gist_id}
https://github.com/conceptofmind
conceptofmind
MDQ6VXNlcjI1MjA4MjI4
https://api.github.com/users/conceptofmind/orgs
https://api.github.com/users/conceptofmind/received_events
https://api.github.com/users/conceptofmind/repos
https://api.github.com/users/conceptofmind/starred{/owner}{/repo}
https://api.github.com/users/conceptofmind/subscriptions
User
https://api.github.com/users/conceptofmind
public
NONE
I think this issue is solved here https://discuss.huggingface.co/t/minhash-deduplication/19992/12?u=loubnabnl, this only happens for very large datasets we will update it in CodeParrot code
2022-08-19T15:26:51Z
https://github.com/huggingface/datasets/issues/4782#issuecomment-1220806479
https://api.github.com/repos/huggingface/datasets/issues/4782
IC_kwDODunzps5IxAdP
https://api.github.com/repos/huggingface/datasets/issues/comments/1220806479/reactions
2022-08-19T15:26:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1220806479
https://avatars.githubusercontent.com/u/44069155?v=4
https://api.github.com/users/loubnabnl/events{/privacy}
https://api.github.com/users/loubnabnl/followers
https://api.github.com/users/loubnabnl/following{/other_user}
https://api.github.com/users/loubnabnl/gists{/gist_id}
https://github.com/loubnabnl
loubnabnl
MDQ6VXNlcjQ0MDY5MTU1
https://api.github.com/users/loubnabnl/orgs
https://api.github.com/users/loubnabnl/received_events
https://api.github.com/users/loubnabnl/repos
https://api.github.com/users/loubnabnl/starred{/owner}{/repo}
https://api.github.com/users/loubnabnl/subscriptions
User
https://api.github.com/users/loubnabnl
public
NONE
Hi @loubnabnl,

Yes, the issue is solved in the discussion thread.

I will close this issue.

Thank you again for all of your help.

Enrico
2022-08-20T02:11:53Z
https://github.com/huggingface/datasets/issues/4782#issuecomment-1221208368
https://api.github.com/repos/huggingface/datasets/issues/4782
IC_kwDODunzps5Iyikw
https://api.github.com/repos/huggingface/datasets/issues/comments/1221208368/reactions
2022-08-20T02:11:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1221208368
https://avatars.githubusercontent.com/u/25208228?v=4
https://api.github.com/users/conceptofmind/events{/privacy}
https://api.github.com/users/conceptofmind/followers
https://api.github.com/users/conceptofmind/following{/other_user}
https://api.github.com/users/conceptofmind/gists{/gist_id}
https://github.com/conceptofmind
conceptofmind
MDQ6VXNlcjI1MjA4MjI4
https://api.github.com/users/conceptofmind/orgs
https://api.github.com/users/conceptofmind/received_events
https://api.github.com/users/conceptofmind/repos
https://api.github.com/users/conceptofmind/starred{/owner}{/repo}
https://api.github.com/users/conceptofmind/subscriptions
User
https://api.github.com/users/conceptofmind
public
MEMBER
Thanks @loubnabnl for pointing out the solution to this issue.
2022-08-22T09:46:28Z
https://github.com/huggingface/datasets/issues/4782#issuecomment-1222114457
https://api.github.com/repos/huggingface/datasets/issues/4782
IC_kwDODunzps5I1_yZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1222114457/reactions
2022-08-22T09:46:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1222114457
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset. 

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```