Add the option of saving in parquet instead of arrow 
CONTRIBUTOR
I think [`Dataset.to_parquet`](https://huggingface.co/docs/datasets/v1.10.2/package_reference/main_classes.html#datasets.Dataset.to_parquet) is what you're looking for.

Let me know if I'm wrong 
2024-05-16T20:38:01Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2116135965
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps5-Iawd
https://api.github.com/repos/huggingface/datasets/issues/comments/2116135965/reactions
2024-05-16T20:38:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2116135965
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
NONE
No, it does not save the metadata json.

We have to recode all meta json load/save
with another custome functions.

save_to_disk
and load should have option with
“Parquet” instead of “arrow”

since “arrow” is never user for production 
(only parquet).

Thanks !

> On May 17, 2024, at 5:38, Frédéric Branchaud-Charron ***@***.***> wrote:
> 
> ﻿
> I think Dataset.to_parquet is what you're looking for.
> 
> Let me know if I'm wrong
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

2024-05-17T03:40:02Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2116567765
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps5-KELV
https://api.github.com/repos/huggingface/datasets/issues/comments/2116567765/reactions
2024-05-17T03:40:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2116567765
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
You can use `to_parquet` and `ds.info.write_to_directory()` to save the dataset info
2024-06-13T16:09:19Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2166119271
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6BHFtn
https://api.github.com/repos/huggingface/datasets/issues/comments/2166119271/reactions
2024-06-13T16:09:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2166119271
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok,

What about loading ?

Should we do in 2 steps ?



> On Jun 14, 2024, at 1:09, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> You can use to_parquet and ds.info.write_to_directory() to save the dataset info
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

2024-06-13T17:46:57Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2166432106
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6BISFq
https://api.github.com/repos/huggingface/datasets/issues/comments/2166432106/reactions
2024-06-13T17:46:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2166432106
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
Yes, and there is DatasetInfo.from_directory(). to reload the info
2024-06-14T10:53:52Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2167771927
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6BNZMX
https://api.github.com/repos/huggingface/datasets/issues/comments/2167771927/reactions
2024-06-14T10:53:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2167771927
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Isn’t easier to combine both
into load_dataset and save_dataset
with parquet options.

2) another question,
How can we download large dataset into disk directly without loading all in memory (!)




> On Jun 14, 2024, at 19:54, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> Yes, and there is DatasetInfo.from_directory(). to reload the info
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

2024-06-14T11:01:05Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2167782882
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6BNb3i
https://api.github.com/repos/huggingface/datasets/issues/comments/2167782882/reactions
2024-06-14T11:01:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2167782882
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
MEMBER
`load_dataset` doesn't load the dataset in memory, it progressively writes to disk in Arrow format and then memory maps the Arrow files. This allows to load datasets bigger than memory and without filling your RAM
2024-06-14T15:06:31Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2168236435
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6BPKmT
https://api.github.com/repos/huggingface/datasets/issues/comments/2168236435/reactions
2024-06-14T15:06:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2168236435
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Sure.
How memory map is managed ?
Managed by the OS ?

Why the need of save_dataset() ?



> On Jun 15, 2024, at 0:06, Quentin Lhoest ***@***.***> wrote:
> 
> ﻿
> load_dataset doesn't load the dataset in memory, it progressively writes to disk in Arrow format and then memory maps the Arrow files. This allows to load datasets bigger than memory and without filling your RAM
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

2024-06-14T16:24:30Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2168361247
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6BPpEf
https://api.github.com/repos/huggingface/datasets/issues/comments/2168361247/reactions
2024-06-14T16:24:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2168361247
https://avatars.githubusercontent.com/u/18707623?v=4
https://api.github.com/users/arita37/events{/privacy}
https://api.github.com/users/arita37/followers
https://api.github.com/users/arita37/following{/other_user}
https://api.github.com/users/arita37/gists{/gist_id}
https://github.com/arita37
arita37
MDQ6VXNlcjE4NzA3NjIz
https://api.github.com/users/arita37/orgs
https://api.github.com/users/arita37/received_events
https://api.github.com/users/arita37/repos
https://api.github.com/users/arita37/starred{/owner}{/repo}
https://api.github.com/users/arita37/subscriptions
User
https://api.github.com/users/arita37
public
NONE
can `to_parquet` do auto sharding ?
2025-02-25T10:25:59Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2681484224
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6f1DPA
https://api.github.com/repos/huggingface/datasets/issues/comments/2681484224/reactions
2025-02-25T10:25:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2681484224
https://avatars.githubusercontent.com/u/61398820?v=4
https://api.github.com/users/Adenialzz/events{/privacy}
https://api.github.com/users/Adenialzz/followers
https://api.github.com/users/Adenialzz/following{/other_user}
https://api.github.com/users/Adenialzz/gists{/gist_id}
https://github.com/Adenialzz
Adenialzz
MDQ6VXNlcjYxMzk4ODIw
https://api.github.com/users/Adenialzz/orgs
https://api.github.com/users/Adenialzz/received_events
https://api.github.com/users/Adenialzz/repos
https://api.github.com/users/Adenialzz/starred{/owner}{/repo}
https://api.github.com/users/Adenialzz/subscriptions
User
https://api.github.com/users/Adenialzz
public
MEMBER
Not at the moment, but you can shard yourself using

```python
num_shards = 32
for index in range(num_shards):
    shard = ds.shard(index=index, num_shards=num_shards)
    shard.to_parquet("data-{index:05d}.parquet")
```
2025-02-25T14:09:36Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2682099433
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6f3Zbp
https://api.github.com/repos/huggingface/datasets/issues/comments/2682099433/reactions
2025-02-25T14:09:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2682099433
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks! That's good!
2025-02-25T14:36:04Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2682206786
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6f3zpC
https://api.github.com/repos/huggingface/datasets/issues/comments/2682206786/reactions
2025-02-25T14:36:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2682206786
https://avatars.githubusercontent.com/u/61398820?v=4
https://api.github.com/users/Adenialzz/events{/privacy}
https://api.github.com/users/Adenialzz/followers
https://api.github.com/users/Adenialzz/following{/other_user}
https://api.github.com/users/Adenialzz/gists{/gist_id}
https://github.com/Adenialzz
Adenialzz
MDQ6VXNlcjYxMzk4ODIw
https://api.github.com/users/Adenialzz/orgs
https://api.github.com/users/Adenialzz/received_events
https://api.github.com/users/Adenialzz/repos
https://api.github.com/users/Adenialzz/starred{/owner}{/repo}
https://api.github.com/users/Adenialzz/subscriptions
User
https://api.github.com/users/Adenialzz
public
NONE
@lhoestq 

Related to this, I'm doing processing on a very large dataset and I save it in shards,  when I load the parquet shards it takes time but doesn't load it all in memory and reindexes it as one dataset which is what I want, so I can store in the cloud and its ready for lazy loading in inference.
dataset = datasets.load_dataset(
                "parquet",
                data_files=f"dataset_parquet/*.parquet",
                keep_in_memory=False,
                storage_options=storage_options,
            ) 

But if I call save_to_disk it rematerializes the whole dataset in RAM (blowing up memory usage and causing OOM)? Is there a way to save a very large dataset so it doesn't need to be reindexed like above. I feel this should be possible as once it's loaded it's cached and I can see cached arrow (but it cant be transported).
2025-04-02T21:48:58Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2773813926
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6lVQqm
https://api.github.com/repos/huggingface/datasets/issues/comments/2773813926/reactions
2025-04-02T21:48:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2773813926
https://avatars.githubusercontent.com/u/132676609?v=4
https://api.github.com/users/jamesharrisivi/events{/privacy}
https://api.github.com/users/jamesharrisivi/followers
https://api.github.com/users/jamesharrisivi/following{/other_user}
https://api.github.com/users/jamesharrisivi/gists{/gist_id}
https://github.com/jamesharrisivi
jamesharrisivi
U_kgDOB-h8AQ
https://api.github.com/users/jamesharrisivi/orgs
https://api.github.com/users/jamesharrisivi/received_events
https://api.github.com/users/jamesharrisivi/repos
https://api.github.com/users/jamesharrisivi/starred{/owner}{/repo}
https://api.github.com/users/jamesharrisivi/subscriptions
User
https://api.github.com/users/jamesharrisivi
public
MEMBER
`save_to_disk` iterates on chunks from the data to write them to disk shard by shard, it doesn't load the dataset in memory.

Maybe try to lower the `max_shard_size` ?
2025-04-03T09:08:11Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2774989047
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6lZvj3
https://api.github.com/repos/huggingface/datasets/issues/comments/2774989047/reactions
2025-04-03T09:08:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2774989047
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok, is it a bug with my particular dataset. 
I do max_shard_size="500Mb" but still the RAM usage starts increasing to the full dataset even though the .arrow chunks are 500mb.
This also happens at the end of my map function (which maps to a much larger number of rows), during tqdm the % until 100 the RAM is constant then rematerialize.
I know it must be possible as if i do 
dataset = datasets.load_dataset(
"parquet",
data_files=f"dataset_parquet/*.parquet",
keep_in_memory=False,
storage_options=storage_options,
)
it takes a long time but the second time it is fast, and no RAM increase, but I know I cant transfer the cached folder for a production inference server.
2025-04-03T11:55:33Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2775518336
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6lbwyA
https://api.github.com/repos/huggingface/datasets/issues/comments/2775518336/reactions
2025-04-03T11:59:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2775518336
https://avatars.githubusercontent.com/u/132676609?v=4
https://api.github.com/users/jamesharrisivi/events{/privacy}
https://api.github.com/users/jamesharrisivi/followers
https://api.github.com/users/jamesharrisivi/following{/other_user}
https://api.github.com/users/jamesharrisivi/gists{/gist_id}
https://github.com/jamesharrisivi
jamesharrisivi
U_kgDOB-h8AQ
https://api.github.com/users/jamesharrisivi/orgs
https://api.github.com/users/jamesharrisivi/received_events
https://api.github.com/users/jamesharrisivi/repos
https://api.github.com/users/jamesharrisivi/starred{/owner}{/repo}
https://api.github.com/users/jamesharrisivi/subscriptions
User
https://api.github.com/users/jamesharrisivi
public
MEMBER
For `map()` it works a similar way: it writes batch by batch. You can lower the `writer_batch_size=` in `map()` (default is 1000)
Feel free to open a new issue to discuss RAM usage in `save_to_disk()` and `map()`, ideally if you can provide a minimal code that reproduces your issue
2025-04-03T12:16:50Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2775603154
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6lcFfS
https://api.github.com/repos/huggingface/datasets/issues/comments/2775603154/reactions
2025-04-03T12:16:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2775603154
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
OK, could it be because I'm loading the initial dataset from GCS?

I understand how it should work  i.e. not balloon from 800mb to 15gb when call .save_to_disk or at end of map. Its loading the full dataset/rematerializing, and not respecting the shard_size or writer_batch_size. When i use shard_size 100mb, the shards are indeed 100mg, just peak memory is the full dataset.

Ok I'd have to create a very large public dataset to recreate it. 
2025-04-03T12:59:13Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2775710928
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6lcfzQ
https://api.github.com/repos/huggingface/datasets/issues/comments/2775710928/reactions
2025-04-03T12:59:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2775710928
https://avatars.githubusercontent.com/u/132676609?v=4
https://api.github.com/users/jamesharrisivi/events{/privacy}
https://api.github.com/users/jamesharrisivi/followers
https://api.github.com/users/jamesharrisivi/following{/other_user}
https://api.github.com/users/jamesharrisivi/gists{/gist_id}
https://github.com/jamesharrisivi
jamesharrisivi
U_kgDOB-h8AQ
https://api.github.com/users/jamesharrisivi/orgs
https://api.github.com/users/jamesharrisivi/received_events
https://api.github.com/users/jamesharrisivi/repos
https://api.github.com/users/jamesharrisivi/starred{/owner}{/repo}
https://api.github.com/users/jamesharrisivi/subscriptions
User
https://api.github.com/users/jamesharrisivi
public
NONE
I ended up manually sharding chunks to disk and using 
```
datasets.load_dataset(
                "parquet",
                data_files=f"{dataset_path}/*.parquet")
```
This is less than ideal as it means it is very slow loading/indexing the first time. But the RSS remains as expected, only a bit larger than the chunks. My use case is I have one k8s service preparing the large dataset, and another to run inference.

I think it could be related to this: https://stasosphere.com/entrepreneur-being/301-mmap-memory-leak-investigation/
The RSS usage still looks like its growing and growing even though its evictable.. And k8s monitors it so crashes the pod?
@lhoestq Have you seen this problem before of using large datasets in production.

2025-05-15T14:03:05Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2883934439
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6r5Vjn
https://api.github.com/repos/huggingface/datasets/issues/comments/2883934439/reactions
2025-05-15T14:03:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2883934439
https://avatars.githubusercontent.com/u/132676609?v=4
https://api.github.com/users/jamesharrisivi/events{/privacy}
https://api.github.com/users/jamesharrisivi/followers
https://api.github.com/users/jamesharrisivi/following{/other_user}
https://api.github.com/users/jamesharrisivi/gists{/gist_id}
https://github.com/jamesharrisivi
jamesharrisivi
U_kgDOB-h8AQ
https://api.github.com/users/jamesharrisivi/orgs
https://api.github.com/users/jamesharrisivi/received_events
https://api.github.com/users/jamesharrisivi/repos
https://api.github.com/users/jamesharrisivi/starred{/owner}{/repo}
https://api.github.com/users/jamesharrisivi/subscriptions
User
https://api.github.com/users/jamesharrisivi
public
MEMBER
It's also possible to encounter that issue if you're using a special kind of disk that has a caching mechanism that loads memory mapped files in RAM.
2025-05-19T12:14:13Z
https://github.com/huggingface/datasets/issues/6903#issuecomment-2890788687
https://api.github.com/repos/huggingface/datasets/issues/6903
IC_kwDODunzps6sTe9P
https://api.github.com/repos/huggingface/datasets/issues/comments/2890788687/reactions
2025-05-19T12:14:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2890788687
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)



### Motivation

because arrow is not used for Production Big data.... (only parquet)

### Your contribution

I can do the testing !