Is dataset iterative or not?
MEMBER
Hello !
Could you give more details ?

If you mean iter through one dataset then yes, `Dataset` object does implement the `__iter__` method so you can use 
```python
for example in dataset:
    # do something
```

If you want to iter through several datasets you can first concatenate them
```python
from datasets import concatenate_datasets

new_dataset = concatenate_datasets([dataset1, dataset2])
```
Let me know if this helps !
2020-11-09T09:18:06Z
https://github.com/huggingface/datasets/issues/815#issuecomment-723881199
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyMzg4MTE5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/723881199/reactions
2020-11-09T09:18:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723881199
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi Huggingface/Datasets team,
I want to use the datasets inside Seq2SeqDataset here
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
and there I need to return back each line from the datasets and I am not
sure how to access each line and implement this?
It seems it also has get_item attribute? so I was not sure if this is
iterative dataset? or if this is non-iterable datasets?
thanks.



On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hello !
> Could you give more details ?
>
> If you mean iter through one dataset then yes, Dataset object does
> implement the __iter__ method so you can use
>
> for example in dataset:
>     # do something
>
> If you want to iter through several datasets you can first concatenate them
>
> from datasets import concatenate_datasets
> new_dataset = concatenate_datasets([dataset1, dataset2])
>
> Let me know if this helps !
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
> .
>

2020-11-09T09:23:15Z
https://github.com/huggingface/datasets/issues/815#issuecomment-723884133
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyMzg4NDEzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/723884133/reactions
2020-11-09T09:23:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723884133
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
could you tell me please if datasets also has __getitem__ any idea on how
to integrate it with Seq2SeqDataset is appreciated thanks

On Mon, Nov 9, 2020 at 10:22 AM Rabeeh Karimi Mahabadi <rabeeh@google.com>
wrote:

> Hi Huggingface/Datasets team,
> I want to use the datasets inside Seq2SeqDataset here
> https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
> and there I need to return back each line from the datasets and I am not
> sure how to access each line and implement this?
> It seems it also has get_item attribute? so I was not sure if this is
> iterative dataset? or if this is non-iterable datasets?
> thanks.
>
>
>
> On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
> wrote:
>
>> Hello !
>> Could you give more details ?
>>
>> If you mean iter through one dataset then yes, Dataset object does
>> implement the __iter__ method so you can use
>>
>> for example in dataset:
>>     # do something
>>
>> If you want to iter through several datasets you can first concatenate
>> them
>>
>> from datasets import concatenate_datasets
>> new_dataset = concatenate_datasets([dataset1, dataset2])
>>
>> Let me know if this helps !
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
>> .
>>
>

2020-11-09T09:24:44Z
https://github.com/huggingface/datasets/issues/815#issuecomment-723885025
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyMzg4NTAyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/723885025/reactions
2020-11-09T09:24:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723885025
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
MEMBER
`datasets.Dataset` objects implement indeed `__getitem__`. It returns a dictionary with one field per column.

We've not added the integration of the datasets library for the seq2seq utilities yet. The current seq2seq utilities are based on text files.

However as soon as you have a `datasets.Dataset` with columns "tgt_texts" (str), "src_texts" (str), and "id" (int) you should be able to implement your own Seq2SeqDataset class that wraps your dataset object. Does that make sense to you ?
2020-11-09T10:16:45Z
https://github.com/huggingface/datasets/issues/815#issuecomment-723915556
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyMzkxNTU1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/723915556/reactions
2020-11-09T10:17:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723915556
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi
I am sorry for asking it multiple times but I am not getting the dataloader
type, could you confirm if the dataset library returns back an iterable
type dataloader or a mapping type one where one has access to __getitem__,
in the former case, one can iterate with __iter__, and how I can configure
it to return the data back as the iterative type? I am dealing with
large-scale datasets and  I do not want to bring all in memory
thanks for your help
Best regards
Rabeeh

On Mon, Nov 9, 2020 at 11:17 AM Quentin Lhoest <notifications@github.com>
wrote:

> datasets.Dataset objects implement indeed __getitem__. It returns a
> dictionary with one field per column.
>
> We've not added the integration of the datasets library for the seq2seq
> utilities yet. The current seq2seq utilities are based on text files.
>
> However as soon as you have a datasets.Dataset with columns "tgt_texts"
> (str), "src_texts" (str), and "id" (int) you should be able to implement
> your own Seq2SeqDataset class that wraps your dataset object. Does that
> make sense ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723915556>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYOC22EM7F666BZSOTSO66R3ANCNFSM4TPB7OWA>
> .
>

2020-11-09T17:37:04Z
https://github.com/huggingface/datasets/issues/815#issuecomment-724163811
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyNDE2MzgxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/724163811/reactions
2020-11-09T17:37:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/724163811
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
MEMBER
`datasets.Dataset` objects are both iterative and mapping types: it has both `__iter__` and `__getitem__`
For example you can do
```python
for example in dataset:
    # do something
```
or
```python
for i in range(len(dataset)):
    example = dataset[i]
    # do something
```
When you do that, one and only one example is loaded into memory at a time.
2020-11-09T17:42:59Z
https://github.com/huggingface/datasets/issues/815#issuecomment-724167101
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyNDE2NzEwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/724167101/reactions
2020-11-09T17:42:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/724167101
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi there, 
Here is what I am trying, this is not working for me in map-style datasets, could you please tell me how to use datasets with being able to access ___getitem__ ? could you assist me please correcting this example? I need map-style datasets which is formed from concatenation of two datasets from your library. thanks 


```
import datasets
dataset1 = load_dataset("squad", split="train[:10]")
dataset1 = dataset1.map(lambda example: {"src_texts": "question: {0} context: {1} ".format(
    example["question"], example["context"]),
    "tgt_texts": example["answers"]["text"][0]}, remove_columns=dataset1.column_names)
dataset2 = load_dataset("imdb", split="train[:10]")
dataset2 = dataset2.map(lambda example: {"src_texts": "imdb: " + example["text"],
            "tgt_texts": str(example["label"])}, remove_columns=dataset2.column_names)
train_dataset = datasets.concatenate_datasets([dataset1, dataset2])
train_dataset.set_format(type='torch', columns=['src_texts', 'tgt_texts'])
dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)
for id, batch in enumerate(dataloader):
    print(batch)

```
2020-11-10T10:20:34Z
https://github.com/huggingface/datasets/issues/815#issuecomment-724608710
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyNDYwODcxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/724608710/reactions
2020-11-10T10:20:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/724608710
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
NONE
closed since I found this response on the issue https://github.com/huggingface/datasets/issues/469
2020-11-10T10:50:03Z
https://github.com/huggingface/datasets/issues/815#issuecomment-724624077
https://api.github.com/repos/huggingface/datasets/issues/815
MDEyOklzc3VlQ29tbWVudDcyNDYyNDA3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/724624077/reactions
2020-11-10T10:50:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/724624077
https://avatars.githubusercontent.com/u/73364383?v=4
https://api.github.com/users/rabeehkarimimahabadi/events{/privacy}
https://api.github.com/users/rabeehkarimimahabadi/followers
https://api.github.com/users/rabeehkarimimahabadi/following{/other_user}
https://api.github.com/users/rabeehkarimimahabadi/gists{/gist_id}
https://github.com/rabeehkarimimahabadi
rabeehkarimimahabadi
MDQ6VXNlcjczMzY0Mzgz
https://api.github.com/users/rabeehkarimimahabadi/orgs
https://api.github.com/users/rabeehkarimimahabadi/received_events
https://api.github.com/users/rabeehkarimimahabadi/repos
https://api.github.com/users/rabeehkarimimahabadi/starred{/owner}{/repo}
https://api.github.com/users/rabeehkarimimahabadi/subscriptions
User
https://api.github.com/users/rabeehkarimimahabadi
public
Hi
I want to use your library for large-scale training, I am not sure if this is implemented as iterative datasets or not?
could you provide me with example how I can use datasets as iterative datasets?
thanks