Dataset viewer displays wrong statists
MEMBER
i think this issue is more for https://github.com/huggingface/dataset-viewer
2024-11-13T11:24:51Z
https://github.com/huggingface/datasets/issues/7289#issuecomment-2473294199
https://api.github.com/repos/huggingface/datasets/issues/7289
IC_kwDODunzps6Ta3l3
https://api.github.com/repos/huggingface/datasets/issues/comments/2473294199/reactions
2024-11-13T11:24:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2473294199
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
### Describe the bug

In [my dataset](https://huggingface.co/datasets/speedcell4/opus-unigram2), there is a column called `lang2`, and there are 94 different classes in total, but the viewer says there are 83 values only. This issue only arises in the `train` split. The total number of values is also 94 in the `test` and `dev` columns, viewer tells the correct number of them.

<img width="177" alt="image" src="https://github.com/user-attachments/assets/78d76ef2-fe0e-4fa3-85e0-fb2552813d1c">

### Steps to reproduce the bug

```python3
from datasets import load_dataset

ds = load_dataset('speedcell4/opus-unigram2').unique('lang2')
for key, lang2 in ds.items():
    print(key, len(lang2))
```

This script returns the following and tells that the `train` split has 94 values in the `lang2` column.

```
train 94
dev 94
test 94
zero 5
```

### Expected behavior

94 in the reviewer.

### Environment info

Collecting environment information...
PyTorch version: 2.4.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 8.2.2004 (Core)  (x86_64)
GCC version: (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
Clang version: Could not collect
CMake version: version 3.11.4
Libc version: glibc-2.28

Python version: 3.9.20 (main, Oct  3 2024, 07:27:41)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.18.0-193.28.1.el8_2.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.85.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              64
On-line CPU(s) list: 0-63
Thread(s) per core:  1
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        4
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7542 32-Core Processor
Stepping:            0
CPU MHz:             3389.114
BogoMIPS:            5789.40
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-15
NUMA node1 CPU(s):   16-31
NUMA node2 CPU(s):   32-47
NUMA node3 CPU(s):   48-63
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.4.1+cu121
[pip3] torchaudio==2.4.1+cu121
[pip3] torchdevice==0.1.1
[pip3] torchglyph==0.3.2
[pip3] torchmetrics==1.5.0
[pip3] torchrua==0.5.1
[pip3] torchvision==0.19.1+cu121
[pip3] triton==3.0.0
[pip3] datasets==3.0.1
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] torch                     2.4.1+cu121              pypi_0    pypi
[conda] torchaudio                2.4.1+cu121              pypi_0    pypi
[conda] torchdevice               0.1.1                    pypi_0    pypi
[conda] torchglyph                0.3.2                    pypi_0    pypi
[conda] torchmetrics              1.5.0                    pypi_0    pypi
[conda] torchrua                  0.5.1                    pypi_0    pypi
[conda] torchvision               0.19.1+cu121             pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi