Transformer Class on dataset
MEMBER
Hi ! Do you have an example in mind that shows how this could be useful ?
2021-07-06T08:39:29Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-874573977
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NDU3Mzk3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/874573977/reactions
2021-07-06T08:39:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874573977
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Example:

Merge 2 datasets into one datasets

Label extraction from dataset

dataset(text, label)
   —> dataset(text, newlabel)

TextCleaning.


For image dataset, 
Transformation are easier (ie linear algebra).






> On Jul 6, 2021, at 17:39, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> Hi ! Do you have an example in mind that shows how this could be useful ?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

2021-07-06T09:21:00Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-874603186
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NDYwMzE4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/874603186/reactions
2021-07-06T09:21:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874603186
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
There are already a few transformations that you can apply on a dataset using methods like `dataset.map()`.
You can find examples in the documentation here:
https://huggingface.co/docs/datasets/processing.html

You can merge two datasets with `concatenate_datasets()` or do label extraction with `dataset.map()` for example
2021-07-06T13:00:19Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-874738151
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NDczODE1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/874738151/reactions
2021-07-06T13:00:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874738151
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok, sure.

Thanks for pointing on functional part.
My question is more
“Philosophical”/Design perspective.

There are 2 perspetive:
  Add transformation methods to 
     Dataset Class


 OR Create a Transformer Class
   which operates on Dataset Class.

T(Dataset) —> Dataset

datasetnew = MyTransform.transform(dataset)
datasetNew.save(path)


What would be the difficulty
of implementing a Transformer Class
operating at dataset level ?


thanks









> On Jul 6, 2021, at 22:00, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> There are already a few transformations that you can apply on a dataset using methods like dataset.map().
> You can find examples in the documentation here:
> https://huggingface.co/docs/datasets/processing.html
> 
> You can merge two datasets with concatenate_datasets() or do label extraction with dataset.map() for example
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

2021-07-06T14:22:56Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-874805686
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NDgwNTY4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/874805686/reactions
2021-07-06T14:22:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/874805686
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.

I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
2021-07-07T07:50:58Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-875374643
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NTM3NDY0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/875374643/reactions
2021-07-07T07:50:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875374643
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for reply.

What would be the constraints
to have
Dataset —> Dataset consistency ?

Main issue would be
larger than memory dataset and
serialization on disk.

Technically,
one still process at atomic level
and try to wrap the full results
into Dataset…. (!)

What would you think ?








> On Jul 7, 2021, at 16:51, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> I can imagine that this would be a useful API to implement processing pipelines as transforms. They could be used to perform higher level transforms compared to the atomic transforms allowed by methods like map, filter, etc.
> 
> I guess if you find any transform that could be useful for text dataset processing, image dataset processing etc. we could definitely start having such transforms :)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

2021-07-07T08:41:25Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-875411668
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NTQxMTY2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/875411668/reactions
2021-07-07T08:41:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875411668
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
We can be pretty flexible and not impose any constraints for transforms.

Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like `map` work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
2021-07-07T09:04:08Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-875428156
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NTQyODE1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/875428156/reactions
2021-07-07T09:04:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875428156
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok thanks.

But, Dataset has various flavors.
In current design of Dataset,
   how the serialization on disk is done (?)


The main issue is serialization 
of   newdataset= Transform(Dataset)
 (ie thats why am referring to Out Of memory dataset…):

   Should be part of Transform or part of dataset ?




Maybe, not, since the output is aimed to feed model in memory (?)








> On Jul 7, 2021, at 18:04, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> We can be pretty flexible and not impose any constraints for transforms.
> 
> Moreover, this library is designed to support datasets bigger than memory. The datasets are loaded from the disk via memory mapping, without filling up RAM. Even processing functions like map work in a batched fashion to not fill up your RAM. So this shouldn't be an issue
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

2021-07-08T01:41:16Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-876051780
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NjA1MTc4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/876051780/reactions
2021-07-08T01:41:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/876051780
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
I'm not sure I understand, could you elaborate a bit more please ?

Each dataset is a wrapper of a PyArrow Table that contains all the data. The table is loaded from an arrow file on the disk.
We have an ArrowWriter and ArrowReader class to write/read arrow tables on disk or in in-memory buffers.
2021-07-08T08:22:04Z
https://github.com/huggingface/datasets/issues/2596#issuecomment-876237453
https://api.github.com/repos/huggingface/datasets/issues/2596
MDEyOklzc3VlQ29tbWVudDg3NjIzNzQ1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/876237453/reactions
2021-07-08T08:22:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/876237453
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Just wondering if you have intenttion to create

TransformerClass :
    dataset --> dataset

and make determnistic transformation (ie not fit).





