Update Open Subtitles corpus with original sentence IDs
MEMBER
Hi ! You're right this can can useful.
This should be easy to add, so feel free to give it a try if you want to contribute :)
I think we just need to add it to the _generate_examples method of the OpenSubtitles dataset builder [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L103)
2021-02-10T16:18:25Z
https://github.com/huggingface/datasets/issues/1844#issuecomment-776827773
https://api.github.com/repos/huggingface/datasets/issues/1844
MDEyOklzc3VlQ29tbWVudDc3NjgyNzc3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/776827773/reactions
2021-02-10T16:18:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776827773
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: 
(the following is line `22497315` of the `ids` file of the `de-en` dump)


`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)


Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.


2021-02-10T16:58:07Z
https://github.com/huggingface/datasets/issues/1844#issuecomment-776859888
https://api.github.com/repos/huggingface/datasets/issues/1844
MDEyOklzc3VlQ29tbWVudDc3Njg1OTg4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/776859888/reactions
2021-02-10T16:58:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776859888
https://avatars.githubusercontent.com/u/19476123?v=4
https://api.github.com/users/Valahaar/events{/privacy}
https://api.github.com/users/Valahaar/followers
https://api.github.com/users/Valahaar/following{/other_user}
https://api.github.com/users/Valahaar/gists{/gist_id}
https://github.com/Valahaar
Valahaar
MDQ6VXNlcjE5NDc2MTIz
https://api.github.com/users/Valahaar/orgs
https://api.github.com/users/Valahaar/received_events
https://api.github.com/users/Valahaar/repos
https://api.github.com/users/Valahaar/starred{/owner}{/repo}
https://api.github.com/users/Valahaar/subscriptions
User
https://api.github.com/users/Valahaar
public
MEMBER
I like the idea of having `year`, `imdbId` and `subtitleId` as columns for filtering for example.
And for the `sentenceIds` a list of integers is fine.
2021-02-10T17:11:37Z
https://github.com/huggingface/datasets/issues/1844#issuecomment-776869302
https://api.github.com/repos/huggingface/datasets/issues/1844
MDEyOklzc3VlQ29tbWVudDc3Njg2OTMwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/776869302/reactions
2021-02-10T17:11:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776869302
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for improving it @Valahaar :) 
2021-02-10T17:35:09Z
https://github.com/huggingface/datasets/issues/1844#issuecomment-776885023
https://api.github.com/repos/huggingface/datasets/issues/1844
MDEyOklzc3VlQ29tbWVudDc3Njg4NTAyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/776885023/reactions
2021-02-10T17:35:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776885023
https://avatars.githubusercontent.com/u/1183441?v=4
https://api.github.com/users/abhishekkrthakur/events{/privacy}
https://api.github.com/users/abhishekkrthakur/followers
https://api.github.com/users/abhishekkrthakur/following{/other_user}
https://api.github.com/users/abhishekkrthakur/gists{/gist_id}
https://github.com/abhishekkrthakur
abhishekkrthakur
MDQ6VXNlcjExODM0NDE=
https://api.github.com/users/abhishekkrthakur/orgs
https://api.github.com/users/abhishekkrthakur/received_events
https://api.github.com/users/abhishekkrthakur/repos
https://api.github.com/users/abhishekkrthakur/starred{/owner}{/repo}
https://api.github.com/users/abhishekkrthakur/subscriptions
User
https://api.github.com/users/abhishekkrthakur
public
CONTRIBUTOR
Something like this? (adapted from [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L114))

```python
result = (
  sentence_counter,
    {
      "id": str(sentence_counter),
      "meta": {
        "year": year,
        "imdbId": imdb_id,
        "subtitleId": {l1: l1_sub_id, l2: l2_sub_id},
        "sentenceIds": {l1: [... source_sids ...], l2: [... target_sids ...]},
        # or maybe src/tgt? I'd go with the first one for consistency with 'translation'
        "subtitleId": {"src": l1_sub_id, "tgt": l2_sub_id},
        "sentenceIds": {"src": [... source_sids ...], "tgt": [... target_sids ...]},
      },
      "translation": {l1: x, l2: y},
    },
  )
```
Or at top level, avoiding nesting into 'meta'?
2021-02-10T17:39:47Z
https://github.com/huggingface/datasets/issues/1844#issuecomment-776888095
https://api.github.com/repos/huggingface/datasets/issues/1844
MDEyOklzc3VlQ29tbWVudDc3Njg4ODA5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/776888095/reactions
2021-02-10T17:39:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776888095
https://avatars.githubusercontent.com/u/19476123?v=4
https://api.github.com/users/Valahaar/events{/privacy}
https://api.github.com/users/Valahaar/followers
https://api.github.com/users/Valahaar/following{/other_user}
https://api.github.com/users/Valahaar/gists{/gist_id}
https://github.com/Valahaar
Valahaar
MDQ6VXNlcjE5NDc2MTIz
https://api.github.com/users/Valahaar/orgs
https://api.github.com/users/Valahaar/received_events
https://api.github.com/users/Valahaar/repos
https://api.github.com/users/Valahaar/starred{/owner}{/repo}
https://api.github.com/users/Valahaar/subscriptions
User
https://api.github.com/users/Valahaar
public
CONTRIBUTOR
Merged in #1865, closing. Thanks :)
2021-02-12T17:38:58Z
https://github.com/huggingface/datasets/issues/1844#issuecomment-778338452
https://api.github.com/repos/huggingface/datasets/issues/1844
MDEyOklzc3VlQ29tbWVudDc3ODMzODQ1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/778338452/reactions
2021-02-12T17:38:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/778338452
https://avatars.githubusercontent.com/u/19476123?v=4
https://api.github.com/users/Valahaar/events{/privacy}
https://api.github.com/users/Valahaar/followers
https://api.github.com/users/Valahaar/following{/other_user}
https://api.github.com/users/Valahaar/gists{/gist_id}
https://github.com/Valahaar
Valahaar
MDQ6VXNlcjE5NDc2MTIz
https://api.github.com/users/Valahaar/orgs
https://api.github.com/users/Valahaar/received_events
https://api.github.com/users/Valahaar/repos
https://api.github.com/users/Valahaar/starred{/owner}{/repo}
https://api.github.com/users/Valahaar/subscriptions
User
https://api.github.com/users/Valahaar
public
Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks!