Add WIT Dataset
CONTRIBUTOR
@hassiahk is working on it  #2810 
2021-09-13T20:05:15Z
https://github.com/huggingface/datasets/issues/2902#issuecomment-918533192
https://api.github.com/repos/huggingface/datasets/issues/2902
IC_kwDODunzps42v7RI
https://api.github.com/repos/huggingface/datasets/issues/comments/918533192/reactions
2021-09-13T20:05:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/918533192
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
MEMBER
WikiMedia is now hosting the pixel values directly which should make it a lot easier!
The files can be found here:
https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/
https://analytics.wikimedia.org/published/datasets/one-off/caption_competition/training/image_pixels/
2021-09-13T20:08:13Z
https://github.com/huggingface/datasets/issues/2902#issuecomment-918535546
https://api.github.com/repos/huggingface/datasets/issues/2902
IC_kwDODunzps42v716
https://api.github.com/repos/huggingface/datasets/issues/comments/918535546/reactions
2021-09-13T20:08:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/918535546
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
CONTRIBUTOR
> @hassiahk is working on it #2810

Thank you @bhavitvyamalik! Added this issue so we could track progress 😄 . Just linked the PR as well for visibility. 
2021-09-13T21:10:00Z
https://github.com/huggingface/datasets/issues/2902#issuecomment-918579291
https://api.github.com/repos/huggingface/datasets/issues/2902
IC_kwDODunzps42wGhb
https://api.github.com/repos/huggingface/datasets/issues/comments/918579291/reactions
2021-09-13T21:10:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/918579291
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
CONTRIBUTOR
Hey folks, we are now hosting the merged pixel values + embeddings + metadata ourselves. I gave it a try - [nateraw/wit](https://huggingface.co/datasets/nateraw/wit)

**⚠️ - Make sure you add `streaming=True` unless you're prepared to download 400GB of data!**

```python
from datasets import load_dataset

ds = load_dataset('nateraw/wit', streaming=True)
example = next(iter(ds))
```

```python
>>> example = next(iter(ds['train']))
>>> example.keys()
dict_keys(['b64_bytes', 'original_width', 'image_url', 'wit_features', 'original_height', 'metadata_url', 'mime_type', 'caption_attribution_description', 'embedding'])
>>> example['wit_features'].keys()
dict_keys(['hierarchical_section_title', 'language', 'attribution_passes_lang_id', 'context_section_description', 'is_main_image', 'page_title', 'caption_title_and_reference_description', 'caption_alt_text_description', 'caption_reference_description', 'page_url', 'context_page_description', 'section_title', 'page_changed_recently'])
```
2021-09-27T17:46:55Z
https://github.com/huggingface/datasets/issues/2902#issuecomment-928107694
https://api.github.com/repos/huggingface/datasets/issues/2902
IC_kwDODunzps43Ucyu
https://api.github.com/repos/huggingface/datasets/issues/comments/928107694/reactions
2021-09-27T17:46:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/928107694
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
COLLABORATOR
Hi! `datasets` now hosts two versions of the WIT dataset:
* [`google/wit`](https://huggingface.co/datasets/google/wit): Google's version with the image URLs
* [`wikimedia/wit_base`](https://huggingface.co/datasets/wikimedia/wit_base): Wikimedia's version with the images + ResNet embeddings, but with less data than Google's version
2022-06-01T17:28:32Z
https://github.com/huggingface/datasets/issues/2902#issuecomment-1143919845
https://api.github.com/repos/huggingface/datasets/issues/2902
IC_kwDODunzps5ELtTl
https://api.github.com/repos/huggingface/datasets/issues/comments/1143919845/reactions
2022-06-01T17:28:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1143919845
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Does this dataset work with `data_files` parameter. I tried and it doesn't. it attempts to download ~600 GB data even when i specify single file
2024-10-02T15:37:47Z
https://github.com/huggingface/datasets/issues/2902#issuecomment-2388996242
https://api.github.com/repos/huggingface/datasets/issues/2902
IC_kwDODunzps6OZTCS
https://api.github.com/repos/huggingface/datasets/issues/comments/2388996242/reactions
2024-10-02T15:37:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2388996242
https://avatars.githubusercontent.com/u/25535309?v=4
https://api.github.com/users/databaaz/events{/privacy}
https://api.github.com/users/databaaz/followers
https://api.github.com/users/databaaz/following{/other_user}
https://api.github.com/users/databaaz/gists{/gist_id}
https://github.com/databaaz
databaaz
MDQ6VXNlcjI1NTM1MzA5
https://api.github.com/users/databaaz/orgs
https://api.github.com/users/databaaz/received_events
https://api.github.com/users/databaaz/repos
https://api.github.com/users/databaaz/starred{/owner}{/repo}
https://api.github.com/users/databaaz/subscriptions
User
https://api.github.com/users/databaaz
public
## Adding a Dataset
- **Name:** *WIT*
- **Description:** *Wikipedia-based Image Text Dataset*
- **Paper:** *[WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
](https://arxiv.org/abs/2103.01913)*
- **Data:** *https://github.com/google-research-datasets/wit*
- **Motivation:**  (excerpt from their Github README.md)

> - The largest multimodal dataset (publicly available at the time of this writing) by the number of image-text examples.
> - A massively multilingual dataset (first of its kind) with coverage for over 100+ languages.
> - A collection of diverse set of concepts and real world entities.
> - Brings forth challenging real-world test sets.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
