[chinese wwm] load_datasets behavior not as expected when using run_mlm_wwm.py script
NONE
@LysandreJik not so sure who to @
Could you help?
2021-12-14T11:22:04Z
https://github.com/huggingface/datasets/issues/3411#issuecomment-993441970
https://api.github.com/repos/huggingface/datasets/issues/3411
IC_kwDODunzps47Nriy
https://api.github.com/repos/huggingface/datasets/issues/comments/993441970/reactions
2021-12-14T11:22:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/993441970
https://avatars.githubusercontent.com/u/52968111?v=4
https://api.github.com/users/hyusterr/events{/privacy}
https://api.github.com/users/hyusterr/followers
https://api.github.com/users/hyusterr/following{/other_user}
https://api.github.com/users/hyusterr/gists{/gist_id}
https://github.com/hyusterr
hyusterr
MDQ6VXNlcjUyOTY4MTEx
https://api.github.com/users/hyusterr/orgs
https://api.github.com/users/hyusterr/received_events
https://api.github.com/users/hyusterr/repos
https://api.github.com/users/hyusterr/starred{/owner}{/repo}
https://api.github.com/users/hyusterr/subscriptions
User
https://api.github.com/users/hyusterr
public
MEMBER
Hi @hyusterr, I believe it is @wlhgtc from https://github.com/huggingface/transformers/pull/9887
2021-12-22T11:21:33Z
https://github.com/huggingface/datasets/issues/3411#issuecomment-999497659
https://api.github.com/repos/huggingface/datasets/issues/3411
IC_kwDODunzps47kx-7
https://api.github.com/repos/huggingface/datasets/issues/comments/999497659/reactions
2021-12-22T11:21:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/999497659
https://avatars.githubusercontent.com/u/30755778?v=4
https://api.github.com/users/LysandreJik/events{/privacy}
https://api.github.com/users/LysandreJik/followers
https://api.github.com/users/LysandreJik/following{/other_user}
https://api.github.com/users/LysandreJik/gists{/gist_id}
https://github.com/LysandreJik
LysandreJik
MDQ6VXNlcjMwNzU1Nzc4
https://api.github.com/users/LysandreJik/orgs
https://api.github.com/users/LysandreJik/received_events
https://api.github.com/users/LysandreJik/repos
https://api.github.com/users/LysandreJik/starred{/owner}{/repo}
https://api.github.com/users/LysandreJik/subscriptions
User
https://api.github.com/users/LysandreJik
public
## Describe the bug

Model I am using (Bert, XLNet ...): bert-base-chinese

The problem arises when using:
* [https://github.com/huggingface/transformers/blob/master/examples/research_projects/mlm_wwm/run_mlm_wwm.py] the official example scripts: `rum_mlm_wwm.py`

The tasks I am working on is: pretraining whole word masking with my own dataset and ref.json file
I tried follow the run_mlm_wwm.py procedure to do whole word masking on pretraining task. my file is in .txt form, where one line represents one sample, with `9,264,784` chinese lines in total. the ref.json file is also contains 9,264,784 lines of whole word masking reference data for my chinese corpus. but when I try to adapt the run_mlm_wwm.py script, it shows that somehow after 
`datasets["train"] = load_dataset(...`
`len(datasets["train"])` returns `9,265,365`
then, after `tokenized_datasets = datasets.map(...`
`len(tokenized_datasets["train"])` returns `9,265,279`
I'm really confused and tried to trace code by myself but can't know what happened after a week trial.

I want to know what happened in the `load_dataset()` function and `datasets.map` here and how did I get more lines of data than I input. so I'm here to ask.
## To reproduce
Sorry that I can't provide my data here since it did not belong to me. but I'm sure I remove the blank lines.

## Expected behavior
I expect the code run as it should. but the AssertionError in line 167 keeps raise as the line of reference json and datasets['train'] differs.

Thanks for your patient reading!

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
- Python version: 3.8.10 
- PyArrow version: 3.0.0
