train_dataset does not implement __len__
MEMBER
Thanks for reporting, @v-yunbin.

Could you please give more details, the steps to reproduce the bug, the complete error back trace and the environment information (`datasets-cli env`)?
2023-04-20T04:57:32Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1515713342
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aV_M-
https://api.github.com/repos/huggingface/datasets/issues/comments/1515713342/reactions
2023-04-20T04:57:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515713342
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
this is a detail error info from transformers：
```
Traceback (most recent call last):
  File "finetune.py", line 177, in <module>
    whisper_finetune(traindir,devdir,outdir)
  File "finetune.py", line 161, in whisper_finetune
    trainer = Seq2SeqTrainer(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 56, in __init__
    super().__init__(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer.py", line 567, in __init__
    raise ValueError(
ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.
```

2023-04-20T05:06:18Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1515720004
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aWA1E
https://api.github.com/repos/huggingface/datasets/issues/comments/1515720004/reactions
2023-04-20T05:08:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515720004
https://avatars.githubusercontent.com/u/38179632?v=4
https://api.github.com/users/ben-8543/events{/privacy}
https://api.github.com/users/ben-8543/followers
https://api.github.com/users/ben-8543/following{/other_user}
https://api.github.com/users/ben-8543/gists{/gist_id}
https://github.com/ben-8543
ben-8543
MDQ6VXNlcjM4MTc5NjMy
https://api.github.com/users/ben-8543/orgs
https://api.github.com/users/ben-8543/received_events
https://api.github.com/users/ben-8543/repos
https://api.github.com/users/ben-8543/starred{/owner}{/repo}
https://api.github.com/users/ben-8543/subscriptions
User
https://api.github.com/users/ben-8543
public
MEMBER
How did you create `train_dataset`? The `datasets` library does not appear in your stack trace.

We need more information in order to reproduce the issue...
2023-04-20T05:36:43Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1515741392
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aWGDQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1515741392/reactions
2023-04-20T05:37:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515741392
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
```
def asr_dataset(traindir,devdir):
    we_voice = IterableDatasetDict()
    #we_voice["train"] = load_from_disk(traindir,streaming=True)
    #we_voice["test"]= load_from_disk(devdir,streaming=True)
    we_voice["train"] = load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train",streaming=True)
    #print(load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train"))
    we_voice["test"] = load_dataset("csv",data_files=os.path.join(devdir,"dev.csv"), split="train",streaming=True)
    we_voice = we_voice.remove_columns(["id"])
    we_voice = we_voice.cast_column("audio", Audio(sampling_rate=16000))
    return we_voice

```
2023-04-20T07:51:59Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1515882057
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aWoZJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1515882057/reactions
2023-04-20T07:51:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515882057
https://avatars.githubusercontent.com/u/38179632?v=4
https://api.github.com/users/ben-8543/events{/privacy}
https://api.github.com/users/ben-8543/followers
https://api.github.com/users/ben-8543/following{/other_user}
https://api.github.com/users/ben-8543/gists{/gist_id}
https://github.com/ben-8543
ben-8543
MDQ6VXNlcjM4MTc5NjMy
https://api.github.com/users/ben-8543/orgs
https://api.github.com/users/ben-8543/received_events
https://api.github.com/users/ben-8543/repos
https://api.github.com/users/ben-8543/starred{/owner}{/repo}
https://api.github.com/users/ben-8543/subscriptions
User
https://api.github.com/users/ben-8543
public
MEMBER
As you are using iterable datasets (`streaming=True`), their length is not defined.

You should:
- Either use non-iterable datasets, which have a defined length: use `DatasetDict` and not passing `streaming=True`
- Or pass `args.max_steps` to the `Trainer`
2023-04-20T08:41:41Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1515948789
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aW4r1
https://api.github.com/repos/huggingface/datasets/issues/comments/1515948789/reactions
2023-04-20T08:47:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515948789
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
I don't know how to give a reasonable args.max_steps...........................
2023-04-20T08:58:55Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1515971454
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aW-N-
https://api.github.com/repos/huggingface/datasets/issues/comments/1515971454/reactions
2023-04-20T08:58:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1515971454
https://avatars.githubusercontent.com/u/38179632?v=4
https://api.github.com/users/ben-8543/events{/privacy}
https://api.github.com/users/ben-8543/followers
https://api.github.com/users/ben-8543/following{/other_user}
https://api.github.com/users/ben-8543/gists{/gist_id}
https://github.com/ben-8543
ben-8543
MDQ6VXNlcjM4MTc5NjMy
https://api.github.com/users/ben-8543/orgs
https://api.github.com/users/ben-8543/received_events
https://api.github.com/users/ben-8543/repos
https://api.github.com/users/ben-8543/starred{/owner}{/repo}
https://api.github.com/users/ben-8543/subscriptions
User
https://api.github.com/users/ben-8543
public
MEMBER
Then you should not use streaming.
2023-04-20T10:41:59Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1516107594
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5aXfdK
https://api.github.com/repos/huggingface/datasets/issues/comments/1516107594/reactions
2023-04-20T10:41:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1516107594
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
@albertvillanova I think @v-yunbin, myself, and others might be slightly confused about max_steps and how it differs from num_train_epochs.
2023-05-15T17:06:02Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1548232604
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5cSCec
https://api.github.com/repos/huggingface/datasets/issues/comments/1548232604/reactions
2023-05-15T17:06:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548232604
https://avatars.githubusercontent.com/u/44705759?v=4
https://api.github.com/users/lkurlandski/events{/privacy}
https://api.github.com/users/lkurlandski/followers
https://api.github.com/users/lkurlandski/following{/other_user}
https://api.github.com/users/lkurlandski/gists{/gist_id}
https://github.com/lkurlandski
lkurlandski
MDQ6VXNlcjQ0NzA1NzU5
https://api.github.com/users/lkurlandski/orgs
https://api.github.com/users/lkurlandski/received_events
https://api.github.com/users/lkurlandski/repos
https://api.github.com/users/lkurlandski/starred{/owner}{/repo}
https://api.github.com/users/lkurlandski/subscriptions
User
https://api.github.com/users/lkurlandski
public
NONE
@lkurlandski A **step** is referring to optimizer's update after back propagation, and it's associated with a batch of data. For example, if a dataset contains 64 examples and you have an overall batch size of 4, then an epoch will have 64/4=16 batches. Therefore, in one epoch, you will have 16 optimizer **steps**.
2023-07-19T20:33:13Z
https://github.com/huggingface/datasets/issues/5773#issuecomment-1642715513
https://api.github.com/repos/huggingface/datasets/issues/5773
IC_kwDODunzps5h6dl5
https://api.github.com/repos/huggingface/datasets/issues/comments/1642715513/reactions
2023-07-19T20:33:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1642715513
https://avatars.githubusercontent.com/u/8699465?v=4
https://api.github.com/users/zhangyilun/events{/privacy}
https://api.github.com/users/zhangyilun/followers
https://api.github.com/users/zhangyilun/following{/other_user}
https://api.github.com/users/zhangyilun/gists{/gist_id}
https://github.com/zhangyilun
zhangyilun
MDQ6VXNlcjg2OTk0NjU=
https://api.github.com/users/zhangyilun/orgs
https://api.github.com/users/zhangyilun/received_events
https://api.github.com/users/zhangyilun/repos
https://api.github.com/users/zhangyilun/starred{/owner}{/repo}
https://api.github.com/users/zhangyilun/subscriptions
User
https://api.github.com/users/zhangyilun
public
when train using data  precessored by the datasets, I get follow warning and it leads to that I can not set epoch numbers:
`ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.`