the compression format requested when saving a dataset in json format is not respected
MEMBER
Thanks for reporting @SaulLu.

At first sight I think the problem is caused because `pandas` only takes into account the `compression` parameter if called with a non-null file path or buffer. And in our implementation, we call pandas `to_json` with `None` `path_or_buf`.

We should fix this:
- either handling directly the `compression` parameter ourselves
- or refactoring to pass non-null path or buffer to pandas

CC: @lhoestq
2021-12-24T12:26:48Z
https://github.com/huggingface/datasets/issues/3480#issuecomment-1000821005
https://api.github.com/repos/huggingface/datasets/issues/3480
IC_kwDODunzps47p1EN
https://api.github.com/repos/huggingface/datasets/issues/comments/1000821005/reactions
2021-12-24T14:05:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1000821005
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
I was thinking if we can handle the `compression` parameter by ourselves? Compression types will be similar to what `pandas` offer. Initially, we can try this with 2-3 compression types and see how good/bad it is? Let me know if it sounds good, I can raise a PR for this next week
2021-12-25T08:15:18Z
https://github.com/huggingface/datasets/issues/3480#issuecomment-1000990102
https://api.github.com/repos/huggingface/datasets/issues/3480
IC_kwDODunzps47qeWW
https://api.github.com/repos/huggingface/datasets/issues/comments/1000990102/reactions
2021-12-25T08:33:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1000990102
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
MEMBER
Hi ! Thanks for your help @bhavitvyamalik :)
Maybe let's start with `gzip` ? I think it's the most common use case, then if we're fine with it we can add other compression methods
2021-12-27T16:25:17Z
https://github.com/huggingface/datasets/issues/3480#issuecomment-1001643846
https://api.github.com/repos/huggingface/datasets/issues/3480
IC_kwDODunzps47s99G
https://api.github.com/repos/huggingface/datasets/issues/comments/1001643846/reactions
2021-12-27T16:25:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1001643846
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
In the documentation of the `to_json` method, it is stated in the parameters that 
> **to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example `compression="gzip"`, the saved file is not compressed.

Would you also have expected compression to be applied? :relaxed: 

## Steps to reproduce the bug
```python
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}
```
### Result with datasets
```python
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz
```
output
```
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}
```
Note: I would expected to see binary data here

### Result with pandas
```python
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz
```
output
```
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���
```
Note: It looks like binary data

## Expected results

I would have expected that the saved result with datasets would also be a binary file

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyArrow version: 5.0.0
