Delta Tables usage using Datasets Library
MEMBER
Hi ! Interesting :) Can you provide concrete examples of cases where it can be useful ?
2022-11-09T13:34:50Z
https://github.com/huggingface/datasets/issues/5219#issuecomment-1308767912
https://api.github.com/repos/huggingface/datasets/issues/5219
IC_kwDODunzps5OAjao
https://api.github.com/repos/huggingface/datasets/issues/comments/1308767912/reactions
2022-11-09T13:34:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1308767912
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Few example blogs and posts that might help on this - 

1. https://hevodata.com/learn/databricks-delta-tables/
2. https://docs.databricks.com/delta/index.html

Basically, we are looking at utility of Datasets library with Delta Lake Tables.

2022-11-18T05:05:27Z
https://github.com/huggingface/datasets/issues/5219#issuecomment-1319552256
https://api.github.com/repos/huggingface/datasets/issues/5219
IC_kwDODunzps5OpsUA
https://api.github.com/repos/huggingface/datasets/issues/comments/1319552256/reactions
2022-11-18T05:05:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1319552256
https://avatars.githubusercontent.com/u/23002137?v=4
https://api.github.com/users/reichenbch/events{/privacy}
https://api.github.com/users/reichenbch/followers
https://api.github.com/users/reichenbch/following{/other_user}
https://api.github.com/users/reichenbch/gists{/gist_id}
https://github.com/reichenbch
reichenbch
MDQ6VXNlcjIzMDAyMTM3
https://api.github.com/users/reichenbch/orgs
https://api.github.com/users/reichenbch/received_events
https://api.github.com/users/reichenbch/repos
https://api.github.com/users/reichenbch/starred{/owner}{/repo}
https://api.github.com/users/reichenbch/subscriptions
User
https://api.github.com/users/reichenbch
public
MEMBER
`datasets` can already read/write from parquet from/to a cloud storage using fsspec, if I understand correctly it's should be possible to load parquet files as delat lake tables no ? :) Or is there someting missing ?
2022-11-18T10:22:25Z
https://github.com/huggingface/datasets/issues/5219#issuecomment-1319807749
https://api.github.com/repos/huggingface/datasets/issues/5219
IC_kwDODunzps5OqqsF
https://api.github.com/repos/huggingface/datasets/issues/comments/1319807749/reactions
2022-11-18T10:22:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1319807749
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq Per my understanding, delta lake table is a bunch of paruqet files together with the meta to support ACID. For example file 1 contains v0.1 of record A while file 2 contains v0.2 of record A. I am assuming the Hugging face dataset would delegate the read/write delta table to 3rd party lib, maybe pyarrow. Correct me if I was wrong @reichenbch 

And I am assuming, people are asking the versioning of Hugging face datasets. But I am assuming Hugging face delegate this function to github and it is not the key requirement for Public Data set. It actually the key function of ML Ops, I am not sure whether hugging face would like expand to that area.
2023-03-02T19:29:11Z
https://github.com/huggingface/datasets/issues/5219#issuecomment-1452438276
https://api.github.com/repos/huggingface/datasets/issues/5219
IC_kwDODunzps5WknME
https://api.github.com/repos/huggingface/datasets/issues/comments/1452438276/reactions
2023-03-02T19:29:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1452438276
https://avatars.githubusercontent.com/u/780760?v=4
https://api.github.com/users/zhenyu/events{/privacy}
https://api.github.com/users/zhenyu/followers
https://api.github.com/users/zhenyu/following{/other_user}
https://api.github.com/users/zhenyu/gists{/gist_id}
https://github.com/zhenyu
zhenyu
MDQ6VXNlcjc4MDc2MA==
https://api.github.com/users/zhenyu/orgs
https://api.github.com/users/zhenyu/received_events
https://api.github.com/users/zhenyu/repos
https://api.github.com/users/zhenyu/starred{/owner}{/repo}
https://api.github.com/users/zhenyu/subscriptions
User
https://api.github.com/users/zhenyu
public
### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.