Arrow file is too large when saving vector data
MEMBER
Hi !
The arrow file size is due to the embeddings. Indeed if they're stored as float32 then the total size of the embeddings is

20 000 000 vectors * 768 dimensions * 4 bytes per dimension ~= 60GB

If you want to reduce the size you can consider using quantization for example, or maybe using dimension reduction techniques.

2021-01-04T13:49:07Z
https://github.com/huggingface/datasets/issues/1662#issuecomment-753984803
https://api.github.com/repos/huggingface/datasets/issues/1662
MDEyOklzc3VlQ29tbWVudDc1Mzk4NDgwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/753984803/reactions
2021-01-04T13:50:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/753984803
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for your reply @lhoestq.
I want to save original embedding for these sentences for subsequent calculations. So does arrow have a way to save in a compressed format to reduce the size of the file?
2021-01-18T03:05:29Z
https://github.com/huggingface/datasets/issues/1662#issuecomment-761949612
https://api.github.com/repos/huggingface/datasets/issues/1662
MDEyOklzc3VlQ29tbWVudDc2MTk0OTYxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/761949612/reactions
2021-01-18T03:06:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/761949612
https://avatars.githubusercontent.com/u/22360336?v=4
https://api.github.com/users/weiwangorg/events{/privacy}
https://api.github.com/users/weiwangorg/followers
https://api.github.com/users/weiwangorg/following{/other_user}
https://api.github.com/users/weiwangorg/gists{/gist_id}
https://github.com/weiwangorg
weiwangorg
MDQ6VXNlcjIyMzYwMzM2
https://api.github.com/users/weiwangorg/orgs
https://api.github.com/users/weiwangorg/received_events
https://api.github.com/users/weiwangorg/repos
https://api.github.com/users/weiwangorg/starred{/owner}{/repo}
https://api.github.com/users/weiwangorg/subscriptions
User
https://api.github.com/users/weiwangorg
public
MEMBER
Arrow doesn't have compression since it is designed to have no serialization overhead
2021-01-18T11:07:31Z
https://github.com/huggingface/datasets/issues/1662#issuecomment-762176103
https://api.github.com/repos/huggingface/datasets/issues/1662
MDEyOklzc3VlQ29tbWVudDc2MjE3NjEwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/762176103/reactions
2021-01-18T11:07:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/762176103
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I see. Thank you.
2021-01-21T14:12:39Z
https://github.com/huggingface/datasets/issues/1662#issuecomment-764669375
https://api.github.com/repos/huggingface/datasets/issues/1662
MDEyOklzc3VlQ29tbWVudDc2NDY2OTM3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/764669375/reactions
2021-01-21T14:12:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/764669375
https://avatars.githubusercontent.com/u/22360336?v=4
https://api.github.com/users/weiwangorg/events{/privacy}
https://api.github.com/users/weiwangorg/followers
https://api.github.com/users/weiwangorg/following{/other_user}
https://api.github.com/users/weiwangorg/gists{/gist_id}
https://github.com/weiwangorg
weiwangorg
MDQ6VXNlcjIyMzYwMzM2
https://api.github.com/users/weiwangorg/orgs
https://api.github.com/users/weiwangorg/received_events
https://api.github.com/users/weiwangorg/repos
https://api.github.com/users/weiwangorg/starred{/owner}{/repo}
https://api.github.com/users/weiwangorg/subscriptions
User
https://api.github.com/users/weiwangorg
public
I computed the sentence embedding of each sentence of bookcorpus data using bert base and saved them to disk. I used 20M sentences and the obtained arrow file is about 59GB while the original text file is only about 1.3GB. Are there any ways to reduce the size of the arrow file?