load_from_disk takes a long time to load local dataset
MEMBER
Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?
2021-03-23T09:46:41Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804763933
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDc2MzkzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/804763933/reactions
2021-03-23T09:46:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804763933
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])
    

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)
    
    return example
```

2021-03-23T09:55:22Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804769252
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDc2OTI1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/804769252/reactions
2021-03-23T10:00:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804769252
https://avatars.githubusercontent.com/u/15007950?v=4
https://api.github.com/users/samsontmr/events{/privacy}
https://api.github.com/users/samsontmr/followers
https://api.github.com/users/samsontmr/following{/other_user}
https://api.github.com/users/samsontmr/gists{/gist_id}
https://github.com/samsontmr
samsontmr
MDQ6VXNlcjE1MDA3OTUw
https://api.github.com/users/samsontmr/orgs
https://api.github.com/users/samsontmr/received_events
https://api.github.com/users/samsontmr/repos
https://api.github.com/users/samsontmr/starred{/owner}{/repo}
https://api.github.com/users/samsontmr/subscriptions
User
https://api.github.com/users/samsontmr
public
MEMBER
Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?
2021-03-23T10:07:38Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804778075
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDc3ODA3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/804778075/reactions
2021-03-23T10:07:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804778075
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.
2021-03-23T10:08:37Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804778782
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDc3ODc4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/804778782/reactions
2021-03-23T10:09:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804778782
https://avatars.githubusercontent.com/u/15007950?v=4
https://api.github.com/users/samsontmr/events{/privacy}
https://api.github.com/users/samsontmr/followers
https://api.github.com/users/samsontmr/following{/other_user}
https://api.github.com/users/samsontmr/gists{/gist_id}
https://github.com/samsontmr
samsontmr
MDQ6VXNlcjE1MDA3OTUw
https://api.github.com/users/samsontmr/orgs
https://api.github.com/users/samsontmr/received_events
https://api.github.com/users/samsontmr/repos
https://api.github.com/users/samsontmr/starred{/owner}{/repo}
https://api.github.com/users/samsontmr/subscriptions
User
https://api.github.com/users/samsontmr
public
NONE
Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.
2021-03-23T13:26:57Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804901939
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDkwMTkzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/804901939/reactions
2021-03-23T13:53:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804901939
https://avatars.githubusercontent.com/u/15007950?v=4
https://api.github.com/users/samsontmr/events{/privacy}
https://api.github.com/users/samsontmr/followers
https://api.github.com/users/samsontmr/following{/other_user}
https://api.github.com/users/samsontmr/gists{/gist_id}
https://github.com/samsontmr
samsontmr
MDQ6VXNlcjE1MDA3OTUw
https://api.github.com/users/samsontmr/orgs
https://api.github.com/users/samsontmr/received_events
https://api.github.com/users/samsontmr/repos
https://api.github.com/users/samsontmr/starred{/owner}{/repo}
https://api.github.com/users/samsontmr/subscriptions
User
https://api.github.com/users/samsontmr
public
MEMBER
Yes this is a known issue: #625 
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`
2021-03-23T14:00:19Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804926466
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDkyNjQ2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/804926466/reactions
2021-03-23T14:00:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804926466
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?
2021-03-23T14:02:20Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-804928000
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNDkyODAwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/804928000/reactions
2021-03-23T14:02:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804928000
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.
2021-03-23T17:12:09Z
https://github.com/huggingface/datasets/issues/2099#issuecomment-805077763
https://api.github.com/repos/huggingface/datasets/issues/2099
MDEyOklzc3VlQ29tbWVudDgwNTA3Nzc2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/805077763/reactions
2021-03-23T17:12:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/805077763
https://avatars.githubusercontent.com/u/15007950?v=4
https://api.github.com/users/samsontmr/events{/privacy}
https://api.github.com/users/samsontmr/followers
https://api.github.com/users/samsontmr/following{/other_user}
https://api.github.com/users/samsontmr/gists{/gist_id}
https://github.com/samsontmr
samsontmr
MDQ6VXNlcjE1MDA3OTUw
https://api.github.com/users/samsontmr/orgs
https://api.github.com/users/samsontmr/received_events
https://api.github.com/users/samsontmr/repos
https://api.github.com/users/samsontmr/starred{/owner}{/repo}
https://api.github.com/users/samsontmr/subscriptions
User
https://api.github.com/users/samsontmr
public
I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)