Add a `reduce` method
COLLABORATOR
Hi! Sure, feel free to open a PR, so we can see the API you have in mind.
2023-02-03T14:11:32Z
https://github.com/huggingface/datasets/issues/5496#issuecomment-1415925307
https://api.github.com/repos/huggingface/datasets/issues/5496
IC_kwDODunzps5UZU47
https://api.github.com/repos/huggingface/datasets/issues/comments/1415925307/reactions
2023-02-03T14:11:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1415925307
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
I would like to give it a go! #self-assign
2023-02-13T15:02:54Z
https://github.com/huggingface/datasets/issues/5496#issuecomment-1428092992
https://api.github.com/repos/huggingface/datasets/issues/5496
IC_kwDODunzps5VHvhA
https://api.github.com/repos/huggingface/datasets/issues/comments/1428092992/reactions
2023-02-13T15:02:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1428092992
https://avatars.githubusercontent.com/u/38854604?v=4
https://api.github.com/users/AJDERS/events{/privacy}
https://api.github.com/users/AJDERS/followers
https://api.github.com/users/AJDERS/following{/other_user}
https://api.github.com/users/AJDERS/gists{/gist_id}
https://github.com/AJDERS
AJDERS
MDQ6VXNlcjM4ODU0NjA0
https://api.github.com/users/AJDERS/orgs
https://api.github.com/users/AJDERS/received_events
https://api.github.com/users/AJDERS/repos
https://api.github.com/users/AJDERS/starred{/owner}{/repo}
https://api.github.com/users/AJDERS/subscriptions
User
https://api.github.com/users/AJDERS
public
COLLABORATOR
Closing as `Dataset.map` can be used instead (see https://github.com/huggingface/datasets/pull/5533#issuecomment-1440571658 and https://github.com/huggingface/datasets/pull/5533#issuecomment-1446403263)
2023-07-21T14:24:32Z
https://github.com/huggingface/datasets/issues/5496#issuecomment-1645676411
https://api.github.com/repos/huggingface/datasets/issues/5496
IC_kwDODunzps5iFwd7
https://api.github.com/repos/huggingface/datasets/issues/comments/1645676411/reactions
2023-07-21T14:24:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1645676411
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hello, is it possible for this issue/PR to be revisited? The problem with the alternatives presented (besides multiple map stages) is that they don't use the cache. A reduce operation is just as expensive as a map operation because it also goes over the entire dataset. It's equally worth caching.

Personally, I have a situation where I would need this and map is far from ideal. I'm working on updating a project of mine to use Huggingface Datasets, and I need to port the loop at https://github.com/colonelwatch/abstracts-search/blob/b90f31ee4cc6e394f829d3a6d9d0311ca390ada9/train.py#L112-L138. Please forgive the code style, here's what it does in English. I have a dataset of about 95 million embeddings, out of which 16384 is taken as a "query" set. For each embedding in the query set, I need to find the ten closest neighbors. These nearest neighbors are used to tune the parameters of a faiss index. The solution is to set up an "accumulator" comprising of the ten closest so far and their distances, then do a single scan over the 95 million (memmapped), then save the results of the "accumulator" for when I want to prototype another index.

The closest approximation to this is multiple map stages, but with such a large "accumulator" having the RAM to do a big batch size becomes critical. At a batch size of 1000, the intermediate accumulators would in theory be about 120 GB! That can be more if I want higher precision than float32. It would already be about the same size as the original embeddings. Using larger batch sizes puts strain on the RAM because I'd be dealing with batch_size x 16384 distances. The best I'd gotten with my RAM, single-threaded, was 65536, and for speed I had to use that thread to feed a GPU. It'd be better if I could use multiple threads to get high throughput instead, or even do all the work in CPU, but to fit the threads I'd need the batch size to be smaller.

All of this intermediate memory could be eliminated if there was a reduce operation.
2024-11-12T00:04:26Z
https://github.com/huggingface/datasets/issues/5496#issuecomment-2469316706
https://api.github.com/repos/huggingface/datasets/issues/5496
IC_kwDODunzps6TLshi
https://api.github.com/repos/huggingface/datasets/issues/comments/2469316706/reactions
2024-11-12T05:58:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2469316706
https://avatars.githubusercontent.com/u/53552802?v=4
https://api.github.com/users/colonelwatch/events{/privacy}
https://api.github.com/users/colonelwatch/followers
https://api.github.com/users/colonelwatch/following{/other_user}
https://api.github.com/users/colonelwatch/gists{/gist_id}
https://github.com/colonelwatch
colonelwatch
MDQ6VXNlcjUzNTUyODAy
https://api.github.com/users/colonelwatch/orgs
https://api.github.com/users/colonelwatch/received_events
https://api.github.com/users/colonelwatch/repos
https://api.github.com/users/colonelwatch/starred{/owner}{/repo}
https://api.github.com/users/colonelwatch/subscriptions
User
https://api.github.com/users/colonelwatch
public
### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`. 

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.  

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend. 