[Feature request] Indexing datasets by a customly-defined id field to enable random access dataset items via the id
MEMBER
You can simply use a python dict as index:

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("BeIR/dbpedia-entity", "corpus", split="corpus")
>>> index = {key: idx for idx, key in enumerate(ds["_id"])}
>>> ds[index["<dbpedia:Pikachu>"]]
{'_id': '<dbpedia:Pikachu>',
 'title': 'Pikachu',
 'text': 'Pikachu (Japanese: ピカチュウ) are a fictional species of Pokémon.  Pokémon are fictional creatures that appear in an assortment of comic books, animated movies and television shows, video games, and trading card games licensed by The Pokémon Company, a Japanese corporation. The Pikachu design was conceived by Ken Sugimori.'}
```
2024-01-02T13:42:51Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-1874042115
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps5vs50D
https://api.github.com/repos/huggingface/datasets/issues/comments/1874042115/reactions
2024-01-02T13:42:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874042115
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for your reply. Yes, I can do that, but it is time-consuming to do that every time I launch the program (some datasets are extremely big). HF Datasets has a nice feature to support instant data loading and efficient random access via row ids. I'm curious if this beneficial feature could be further extended to custom data columns.

2024-01-02T13:52:04Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-1874052172
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps5vs8RM
https://api.github.com/repos/huggingface/datasets/issues/comments/1874052172/reactions
2024-01-02T13:52:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874052172
https://avatars.githubusercontent.com/u/3377221?v=4
https://api.github.com/users/Yu-Shi/events{/privacy}
https://api.github.com/users/Yu-Shi/followers
https://api.github.com/users/Yu-Shi/following{/other_user}
https://api.github.com/users/Yu-Shi/gists{/gist_id}
https://github.com/Yu-Shi
Yu-Shi
MDQ6VXNlcjMzNzcyMjE=
https://api.github.com/users/Yu-Shi/orgs
https://api.github.com/users/Yu-Shi/received_events
https://api.github.com/users/Yu-Shi/repos
https://api.github.com/users/Yu-Shi/starred{/owner}{/repo}
https://api.github.com/users/Yu-Shi/subscriptions
User
https://api.github.com/users/Yu-Shi
public
NONE
+1 on the issue I think it would be extremely useful
2024-04-05T07:55:55Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2039185093
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps55i37F
https://api.github.com/repos/huggingface/datasets/issues/comments/2039185093/reactions
2024-04-05T07:55:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2039185093
https://avatars.githubusercontent.com/u/20661461?v=4
https://api.github.com/users/davidmrau/events{/privacy}
https://api.github.com/users/davidmrau/followers
https://api.github.com/users/davidmrau/following{/other_user}
https://api.github.com/users/davidmrau/gists{/gist_id}
https://github.com/davidmrau
davidmrau
MDQ6VXNlcjIwNjYxNDYx
https://api.github.com/users/davidmrau/orgs
https://api.github.com/users/davidmrau/received_events
https://api.github.com/users/davidmrau/repos
https://api.github.com/users/davidmrau/starred{/owner}{/repo}
https://api.github.com/users/davidmrau/subscriptions
User
https://api.github.com/users/davidmrau
public
NONE
+1. This could be very useful.
2024-06-04T14:43:31Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2147719218
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6AA5gy
https://api.github.com/repos/huggingface/datasets/issues/comments/2147719218/reactions
2024-06-04T14:43:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2147719218
https://avatars.githubusercontent.com/u/47048214?v=4
https://api.github.com/users/GeeYangML/events{/privacy}
https://api.github.com/users/GeeYangML/followers
https://api.github.com/users/GeeYangML/following{/other_user}
https://api.github.com/users/GeeYangML/gists{/gist_id}
https://github.com/GeeYangML
GeeYangML
MDQ6VXNlcjQ3MDQ4MjE0
https://api.github.com/users/GeeYangML/orgs
https://api.github.com/users/GeeYangML/received_events
https://api.github.com/users/GeeYangML/repos
https://api.github.com/users/GeeYangML/starred{/owner}{/repo}
https://api.github.com/users/GeeYangML/subscriptions
User
https://api.github.com/users/GeeYangML
public
NONE
+1 - currently having to manually implement this
2024-06-05T07:54:52Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2149128790
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6AGRpW
https://api.github.com/repos/huggingface/datasets/issues/comments/2149128790/reactions
2024-06-05T07:54:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2149128790
https://avatars.githubusercontent.com/u/46177345?v=4
https://api.github.com/users/ruth-ann/events{/privacy}
https://api.github.com/users/ruth-ann/followers
https://api.github.com/users/ruth-ann/following{/other_user}
https://api.github.com/users/ruth-ann/gists{/gist_id}
https://github.com/ruth-ann
ruth-ann
MDQ6VXNlcjQ2MTc3MzQ1
https://api.github.com/users/ruth-ann/orgs
https://api.github.com/users/ruth-ann/received_events
https://api.github.com/users/ruth-ann/repos
https://api.github.com/users/ruth-ann/starred{/owner}{/repo}
https://api.github.com/users/ruth-ann/subscriptions
User
https://api.github.com/users/ruth-ann
public
NONE
If anyone has an idea how to do this in the right way (perhaps @albertvillanova ?) I would be happy to implement it
2024-08-03T10:58:18Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2266675567
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6HGrlv
https://api.github.com/repos/huggingface/datasets/issues/comments/2266675567/reactions
2024-08-03T11:00:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2266675567
https://avatars.githubusercontent.com/u/20661461?v=4
https://api.github.com/users/davidmrau/events{/privacy}
https://api.github.com/users/davidmrau/followers
https://api.github.com/users/davidmrau/following{/other_user}
https://api.github.com/users/davidmrau/gists{/gist_id}
https://github.com/davidmrau
davidmrau
MDQ6VXNlcjIwNjYxNDYx
https://api.github.com/users/davidmrau/orgs
https://api.github.com/users/davidmrau/received_events
https://api.github.com/users/davidmrau/repos
https://api.github.com/users/davidmrau/starred{/owner}{/repo}
https://api.github.com/users/davidmrau/subscriptions
User
https://api.github.com/users/davidmrau
public
NONE
This would be very helpful to implement aspect ratio bucketing for image and video datasets
2024-10-08T12:50:56Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2399761017
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6PCXJ5
https://api.github.com/repos/huggingface/datasets/issues/comments/2399761017/reactions
2024-10-08T12:50:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2399761017
https://avatars.githubusercontent.com/u/46050679?v=4
https://api.github.com/users/SwayStar123/events{/privacy}
https://api.github.com/users/SwayStar123/followers
https://api.github.com/users/SwayStar123/following{/other_user}
https://api.github.com/users/SwayStar123/gists{/gist_id}
https://github.com/SwayStar123
SwayStar123
MDQ6VXNlcjQ2MDUwNjc5
https://api.github.com/users/SwayStar123/orgs
https://api.github.com/users/SwayStar123/received_events
https://api.github.com/users/SwayStar123/repos
https://api.github.com/users/SwayStar123/starred{/owner}{/repo}
https://api.github.com/users/SwayStar123/subscriptions
User
https://api.github.com/users/SwayStar123
public
NONE
What you're asking for is an index that's provided with the dataset data, and happens to be optimized for your retrieval use case. Those who don't have this problem now need to download more bytes. It's bad for the planet.

If you want to avoid the indexing work, you can serialize the index dict to a file to be loaded on subsequent runs. You might also use sqlite to lazily create and use a db as your index. 


2025-01-08T03:37:41Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2576660706
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6ZlLji
https://api.github.com/repos/huggingface/datasets/issues/comments/2576660706/reactions
2025-01-08T03:37:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2576660706
https://avatars.githubusercontent.com/u/184949?v=4
https://api.github.com/users/gar1t/events{/privacy}
https://api.github.com/users/gar1t/followers
https://api.github.com/users/gar1t/following{/other_user}
https://api.github.com/users/gar1t/gists{/gist_id}
https://github.com/gar1t
gar1t
MDQ6VXNlcjE4NDk0OQ==
https://api.github.com/users/gar1t/orgs
https://api.github.com/users/gar1t/received_events
https://api.github.com/users/gar1t/repos
https://api.github.com/users/gar1t/starred{/owner}{/repo}
https://api.github.com/users/gar1t/subscriptions
User
https://api.github.com/users/gar1t
public
NONE
> If you want to avoid the indexing work, you can serialize the index dict to a file to be loaded on subsequent runs. You might also use sqlite to lazily create and use a db as your index.

Sure, but then can't we have a default implementation that is well optimized and part of the library instead of having a thousand different implementations of this? It does not need to be something that gets downloaded with the dataset (In the same way that when you get the dataset and run map, it does not save and makes people download the mapped version)

2025-01-22T10:59:44Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2606927023
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6bYoyv
https://api.github.com/repos/huggingface/datasets/issues/comments/2606927023/reactions
2025-01-22T10:59:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2606927023
https://avatars.githubusercontent.com/u/155949423?v=4
https://api.github.com/users/carlos-lassance/events{/privacy}
https://api.github.com/users/carlos-lassance/followers
https://api.github.com/users/carlos-lassance/following{/other_user}
https://api.github.com/users/carlos-lassance/gists{/gist_id}
https://github.com/carlos-lassance
carlos-lassance
U_kgDOCUuZbw
https://api.github.com/users/carlos-lassance/orgs
https://api.github.com/users/carlos-lassance/received_events
https://api.github.com/users/carlos-lassance/repos
https://api.github.com/users/carlos-lassance/starred{/owner}{/repo}
https://api.github.com/users/carlos-lassance/subscriptions
User
https://api.github.com/users/carlos-lassance
public
NONE
+1. I wanted to use such a feature.
2025-05-05T13:25:23Z
https://github.com/huggingface/datasets/issues/6532#issuecomment-2851006510
https://api.github.com/repos/huggingface/datasets/issues/6532
IC_kwDODunzps6p7ugu
https://api.github.com/repos/huggingface/datasets/issues/comments/2851006510/reactions
2025-05-05T13:25:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2851006510
https://avatars.githubusercontent.com/u/91313520?v=4
https://api.github.com/users/alicemabille/events{/privacy}
https://api.github.com/users/alicemabille/followers
https://api.github.com/users/alicemabille/following{/other_user}
https://api.github.com/users/alicemabille/gists{/gist_id}
https://github.com/alicemabille
alicemabille
MDQ6VXNlcjkxMzEzNTIw
https://api.github.com/users/alicemabille/orgs
https://api.github.com/users/alicemabille/received_events
https://api.github.com/users/alicemabille/repos
https://api.github.com/users/alicemabille/starred{/owner}{/repo}
https://api.github.com/users/alicemabille/subscriptions
User
https://api.github.com/users/alicemabille
public
### Feature request

Some datasets may contain an id-like field, for example the `id` field in [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and the `_id` field in [BeIR/dbpedia-entity](https://huggingface.co/datasets/BeIR/dbpedia-entity). HF datasets support efficient random access via row, but not via this kinds of id fields. I wonder if it is possible to add support for indexing by a custom "id-like" field to enable random access via such ids. The ids may be numbers or strings.

### Motivation

In some cases, especially during inference/evaluation, I may want to find out the item that has a specified id, defined by the dataset itself.

For example, in a typical re-ranking setting in information retrieval, the user may want to re-rank the set of candidate documents of each query. The input is usually presented in a TREC-style run file, with the following format:

```
<qid> Q0 <docno> <rank> <score> <tag>
```

The re-ranking program should be able to fetch the queries and documents according to the `<qid>` and `<docno>`, which are the original id defined in the query/document datasets. To accomplish this, I have to iterate over the whole HF dataset to get the mapping from real ids to row ids every time I start the program, which is time-consuming. Thus I want HF dataset to provide options for users to index by a custom id column, not by row.

### Your contribution

I'm not an expert in this project and I'm afraid that I'm not able to make contributions on the code.