Why is dataset after tokenization far more larger than the orginal one ?
MEMBER
Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`
2020-06-10T08:59:44Z
https://github.com/huggingface/datasets/issues/258#issuecomment-641858058
https://api.github.com/repos/huggingface/datasets/issues/258
MDEyOklzc3VlQ29tbWVudDY0MTg1ODA1OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/641858058/reactions
2020-06-10T08:59:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641858058
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔
2020-06-10T12:11:44Z
https://github.com/huggingface/datasets/issues/258#issuecomment-641962283
https://api.github.com/repos/huggingface/datasets/issues/258
MDEyOklzc3VlQ29tbWVudDY0MTk2MjI4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/641962283/reactions
2020-06-10T12:11:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641962283
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
MEMBER
Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings.
2020-06-10T12:19:24Z
https://github.com/huggingface/datasets/issues/258#issuecomment-641965987
https://api.github.com/repos/huggingface/datasets/issues/258
MDEyOklzc3VlQ29tbWVudDY0MTk2NTk4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/641965987/reactions
2020-06-10T12:19:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641965987
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
2020-06-10T12:46:29Z
https://github.com/huggingface/datasets/issues/258#issuecomment-641982981
https://api.github.com/repos/huggingface/datasets/issues/258
MDEyOklzc3VlQ29tbWVudDY0MTk4Mjk4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/641982981/reactions
2020-06-10T12:46:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641982981
https://avatars.githubusercontent.com/u/17963619?v=4
https://api.github.com/users/richarddwang/events{/privacy}
https://api.github.com/users/richarddwang/followers
https://api.github.com/users/richarddwang/following{/other_user}
https://api.github.com/users/richarddwang/gists{/gist_id}
https://github.com/richarddwang
richarddwang
MDQ6VXNlcjE3OTYzNjE5
https://api.github.com/users/richarddwang/orgs
https://api.github.com/users/richarddwang/received_events
https://api.github.com/users/richarddwang/repos
https://api.github.com/users/richarddwang/starred{/owner}{/repo}
https://api.github.com/users/richarddwang/subscriptions
User
https://api.github.com/users/richarddwang
public
I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?