We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Endless loop when loading dataset
Hi,
I'm a beginner of OpenNMT-py and I also encounter loop on loading training dataset. I have checked Issues 1079 and 1354 but it still not work. My training set is small and I have not use multiple GPU, just one. Please give me some comments, thank you!
My OpenNMT is the latest version and my os is Ubuntu18.04, here is the parameters I used:

python preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo -src_seq_length 1000 -tgt_seq_length 1000 -dynamic_dict -share_vocab

CUDA_VISIBLE_DEVICES=0 python train.py -data data/demo -save_model demo-model -world_size 1 -gpu_ranks 0 -copy_attn
[2019-05-16 16:35:16,797 INFO]  * src vocab size = 988
[2019-05-16 16:35:16,797 INFO]  * tgt vocab size = 988
[2019-05-16 16:35:16,797 INFO] Building model...
[2019-05-16 16:35:19,271 INFO] NMTModel(
  (encoder): RNNEncoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(988, 500, padding_idx=1)
        )
      )
    )
    (rnn): LSTM(500, 500, num_layers=2, dropout=0.3)
  )
  (decoder): InputFeedRNNDecoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(988, 500, padding_idx=1)
        )
      )
    )
    (dropout): Dropout(p=0.3)
    (rnn): StackedLSTM(
      (dropout): Dropout(p=0.3)
      (layers): ModuleList(
        (0): LSTMCell(1000, 500)
        (1): LSTMCell(500, 500)
      )
    )
    (attn): GlobalAttention(
      (linear_in): Linear(in_features=500, out_features=500, bias=False)
      (linear_out): Linear(in_features=1000, out_features=500, bias=False)
    )
    (copy_attn): GlobalAttention(
      (linear_in): Linear(in_features=500, out_features=500, bias=False)
      (linear_out): Linear(in_features=1000, out_features=500, bias=False)
    )
  )
  (generator): CopyGenerator(
    (linear): Linear(in_features=500, out_features=988, bias=True)
    (linear_copy): Linear(in_features=500, out_features=1, bias=True)
  )
)
[2019-05-16 16:35:19,272 INFO] encoder: 4502000
[2019-05-16 16:35:19,272 INFO] decoder: 7497489
[2019-05-16 16:35:19,272 INFO] * number of parameters: 11999489
[2019-05-16 16:35:19,308 INFO] Starting training on GPU: [0]
[2019-05-16 16:35:19,308 INFO] Start training loop and validate every 10000 steps...
[2019-05-16 16:35:19,330 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,382 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,437 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,498 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,562 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,632 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,705 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,782 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,864 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:19,950 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,041 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,136 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,235 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,339 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,448 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,560 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762
[2019-05-16 16:35:20,678 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762

I have tried using CPU only and default parameters, but this situation also happens. How can I do to solve this problem?




--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `onmt/train_single.py`
Content:
```
1 #!/usr/bin/env python
2 """Training on a single process."""
3 import os
4 
5 import torch
6 
7 from onmt.inputters.inputter import build_dataset_iter, \
8     load_old_vocab, old_style_vocab, MultipleDatasetIterator
9 from onmt.model_builder import build_model
10 from onmt.utils.optimizers import Optimizer
11 from onmt.utils.misc import set_random_seed
12 from onmt.trainer import build_trainer
13 from onmt.models import build_model_saver
14 from onmt.utils.logging import init_logger, logger
15 from onmt.utils.parse import ArgumentParser
16 
17 
18 def _check_save_model_path(opt):
19     save_model_path = os.path.abspath(opt.save_model)
20     model_dirname = os.path.dirname(save_model_path)
21     if not os.path.exists(model_dirname):
22         os.makedirs(model_dirname)
23 
24 
25 def _tally_parameters(model):
26     enc = 0
27     dec = 0
28     for name, param in model.named_parameters():
29         if 'encoder' in name:
30             enc += param.nelement()
31         else:
32             dec += param.nelement()
33     return enc + dec, enc, dec
34 
35 
36 def configure_process(opt, device_id):
37     if device_id >= 0:
38         torch.cuda.set_device(device_id)
39     set_random_seed(opt.seed, device_id >= 0)
40 
41 
42 def main(opt, device_id):
43     # NOTE: It's important that ``opt`` has been validated and updated
44     # at this point.
45     configure_process(opt, device_id)
46     init_logger(opt.log_file)
47     assert len(opt.accum_count) == len(opt.accum_steps), \
48         'Number of accum_count values must match number of accum_steps'
49     # Load checkpoint if we resume from a previous training.
50     if opt.train_from:
51         logger.info('Loading checkpoint from %s' % opt.train_from)
52         checkpoint = torch.load(opt.train_from,
53                                 map_location=lambda storage, loc: storage)
54 
55         model_opt = ArgumentParser.ckpt_model_opts(checkpoint["opt"])
56         ArgumentParser.update_model_opts(model_opt)
57         ArgumentParser.validate_model_opts(model_opt)
58         logger.info('Loading vocab from checkpoint at %s.' % opt.train_from)
59         vocab = checkpoint['vocab']
60     else:
61         checkpoint = None
62         model_opt = opt
63         vocab = torch.load(opt.data + '.vocab.pt')
64 
65     # check for code where vocab is saved instead of fields
66     # (in the future this will be done in a smarter way)
67     if old_style_vocab(vocab):
68         fields = load_old_vocab(
69             vocab, opt.model_type, dynamic_dict=opt.copy_attn)
70     else:
71         fields = vocab
72 
73     # Report src and tgt vocab sizes, including for features
74     for side in ['src', 'tgt']:
75         f = fields[side]
76         try:
77             f_iter = iter(f)
78         except TypeError:
79             f_iter = [(side, f)]
80         for sn, sf in f_iter:
81             if sf.use_vocab:
82                 logger.info(' * %s vocab size = %d' % (sn, len(sf.vocab)))
83 
84     # Build model.
85     model = build_model(model_opt, opt, fields, checkpoint)
86     n_params, enc, dec = _tally_parameters(model)
87     logger.info('encoder: %d' % enc)
88     logger.info('decoder: %d' % dec)
89     logger.info('* number of parameters: %d' % n_params)
90     _check_save_model_path(opt)
91 
92     # Build optimizer.
93     optim = Optimizer.from_opt(model, opt, checkpoint=checkpoint)
94 
95     # Build model saver
96     model_saver = build_model_saver(model_opt, opt, model, fields, optim)
97 
98     trainer = build_trainer(
99         opt, device_id, model, fields, optim, model_saver=model_saver)
100 
101     train_iterables = []
102     for train_id in opt.data_ids:
103         if train_id:
104             shard_base = "train_" + train_id
105         else:
106             shard_base = "train"
107         iterable = build_dataset_iter(shard_base, fields, opt, multi=True)
108         train_iterables.append(iterable)
109 
110     train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)
111 
112     valid_iter = build_dataset_iter(
113         "valid", fields, opt, is_train=False)
114 
115     if len(opt.gpu_ranks):
116         logger.info('Starting training on GPU: %s' % opt.gpu_ranks)
117     else:
118         logger.info('Starting training on CPU, could be very slow')
119     train_steps = opt.train_steps
120     if opt.single_pass and train_steps > 0:
121         logger.warning("Option single_pass is enabled, ignoring train_steps.")
122         train_steps = 0
123     trainer.train(
124         train_iter,
125         train_steps,
126         save_checkpoint_steps=opt.save_checkpoint_steps,
127         valid_iter=valid_iter,
128         valid_steps=opt.valid_steps)
129 
130     if opt.tensorboard:
131         trainer.report_manager.tensorboard_writer.close()
132
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/onmt/train_single.py b/onmt/train_single.py
--- a/onmt/train_single.py
+++ b/onmt/train_single.py
@@ -99,15 +99,14 @@
         opt, device_id, model, fields, optim, model_saver=model_saver)
 
     train_iterables = []
-    for train_id in opt.data_ids:
-        if train_id:
+    if len(opt.data_ids) > 1:
+        for train_id in opt.data_ids:
             shard_base = "train_" + train_id
-        else:
-            shard_base = "train"
-        iterable = build_dataset_iter(shard_base, fields, opt, multi=True)
-        train_iterables.append(iterable)
-
-    train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)
+            iterable = build_dataset_iter(shard_base, fields, opt, multi=True)
+            train_iterables.append(iterable)
+        train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)
+    else:
+        train_iter = build_dataset_iter("train", fields, opt)
 
     valid_iter = build_dataset_iter(
         "valid", fields, opt, is_train=False)

{"golden_diff": "diff --git a/onmt/train_single.py b/onmt/train_single.py\n--- a/onmt/train_single.py\n+++ b/onmt/train_single.py\n@@ -99,15 +99,14 @@\n         opt, device_id, model, fields, optim, model_saver=model_saver)\n \n     train_iterables = []\n-    for train_id in opt.data_ids:\n-        if train_id:\n+    if len(opt.data_ids) > 1:\n+        for train_id in opt.data_ids:\n             shard_base = \"train_\" + train_id\n-        else:\n-            shard_base = \"train\"\n-        iterable = build_dataset_iter(shard_base, fields, opt, multi=True)\n-        train_iterables.append(iterable)\n-\n-    train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)\n+            iterable = build_dataset_iter(shard_base, fields, opt, multi=True)\n+            train_iterables.append(iterable)\n+        train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)\n+    else:\n+        train_iter = build_dataset_iter(\"train\", fields, opt)\n \n     valid_iter = build_dataset_iter(\n         \"valid\", fields, opt, is_train=False)\n", "issue": "Endless loop when loading dataset\nHi,\r\nI'm a beginner of OpenNMT-py and I also encounter loop on loading training dataset. I have checked Issues 1079 and 1354 but it still not work. My training set is small and I have not use multiple GPU, just one. Please give me some comments, thank you!\r\nMy OpenNMT is the latest version and my os is Ubuntu18.04, here is the parameters I used:\r\n\r\npython preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo -src_seq_length 1000 -tgt_seq_length 1000 -dynamic_dict -share_vocab\r\n\r\nCUDA_VISIBLE_DEVICES=0 python train.py -data data/demo -save_model demo-model -world_size 1 -gpu_ranks 0 -copy_attn\r\n[2019-05-16 16:35:16,797 INFO]  * src vocab size = 988\r\n[2019-05-16 16:35:16,797 INFO]  * tgt vocab size = 988\r\n[2019-05-16 16:35:16,797 INFO] Building model...\r\n[2019-05-16 16:35:19,271 INFO] NMTModel(\r\n  (encoder): RNNEncoder(\r\n    (embeddings): Embeddings(\r\n      (make_embedding): Sequential(\r\n        (emb_luts): Elementwise(\r\n          (0): Embedding(988, 500, padding_idx=1)\r\n        )\r\n      )\r\n    )\r\n    (rnn): LSTM(500, 500, num_layers=2, dropout=0.3)\r\n  )\r\n  (decoder): InputFeedRNNDecoder(\r\n    (embeddings): Embeddings(\r\n      (make_embedding): Sequential(\r\n        (emb_luts): Elementwise(\r\n          (0): Embedding(988, 500, padding_idx=1)\r\n        )\r\n      )\r\n    )\r\n    (dropout): Dropout(p=0.3)\r\n    (rnn): StackedLSTM(\r\n      (dropout): Dropout(p=0.3)\r\n      (layers): ModuleList(\r\n        (0): LSTMCell(1000, 500)\r\n        (1): LSTMCell(500, 500)\r\n      )\r\n    )\r\n    (attn): GlobalAttention(\r\n      (linear_in): Linear(in_features=500, out_features=500, bias=False)\r\n      (linear_out): Linear(in_features=1000, out_features=500, bias=False)\r\n    )\r\n    (copy_attn): GlobalAttention(\r\n      (linear_in): Linear(in_features=500, out_features=500, bias=False)\r\n      (linear_out): Linear(in_features=1000, out_features=500, bias=False)\r\n    )\r\n  )\r\n  (generator): CopyGenerator(\r\n    (linear): Linear(in_features=500, out_features=988, bias=True)\r\n    (linear_copy): Linear(in_features=500, out_features=1, bias=True)\r\n  )\r\n)\r\n[2019-05-16 16:35:19,272 INFO] encoder: 4502000\r\n[2019-05-16 16:35:19,272 INFO] decoder: 7497489\r\n[2019-05-16 16:35:19,272 INFO] * number of parameters: 11999489\r\n[2019-05-16 16:35:19,308 INFO] Starting training on GPU: [0]\r\n[2019-05-16 16:35:19,308 INFO] Start training loop and validate every 10000 steps...\r\n[2019-05-16 16:35:19,330 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,382 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,437 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,498 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,562 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,632 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,705 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,782 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,864 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:19,950 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,041 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,136 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,235 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,339 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,448 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,560 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n[2019-05-16 16:35:20,678 INFO] Loading dataset from data/demo.train.0.pt, number of examples: 762\r\n\r\nI have tried using CPU only and default parameters, but this situation also happens. How can I do to solve this problem?\r\n\r\n\r\n\n", "before_files": [{"content": "#!/usr/bin/env python\n\"\"\"Training on a single process.\"\"\"\nimport os\n\nimport torch\n\nfrom onmt.inputters.inputter import build_dataset_iter, \\\n    load_old_vocab, old_style_vocab, MultipleDatasetIterator\nfrom onmt.model_builder import build_model\nfrom onmt.utils.optimizers import Optimizer\nfrom onmt.utils.misc import set_random_seed\nfrom onmt.trainer import build_trainer\nfrom onmt.models import build_model_saver\nfrom onmt.utils.logging import init_logger, logger\nfrom onmt.utils.parse import ArgumentParser\n\n\ndef _check_save_model_path(opt):\n    save_model_path = os.path.abspath(opt.save_model)\n    model_dirname = os.path.dirname(save_model_path)\n    if not os.path.exists(model_dirname):\n        os.makedirs(model_dirname)\n\n\ndef _tally_parameters(model):\n    enc = 0\n    dec = 0\n    for name, param in model.named_parameters():\n        if 'encoder' in name:\n            enc += param.nelement()\n        else:\n            dec += param.nelement()\n    return enc + dec, enc, dec\n\n\ndef configure_process(opt, device_id):\n    if device_id >= 0:\n        torch.cuda.set_device(device_id)\n    set_random_seed(opt.seed, device_id >= 0)\n\n\ndef main(opt, device_id):\n    # NOTE: It's important that ``opt`` has been validated and updated\n    # at this point.\n    configure_process(opt, device_id)\n    init_logger(opt.log_file)\n    assert len(opt.accum_count) == len(opt.accum_steps), \\\n        'Number of accum_count values must match number of accum_steps'\n    # Load checkpoint if we resume from a previous training.\n    if opt.train_from:\n        logger.info('Loading checkpoint from %s' % opt.train_from)\n        checkpoint = torch.load(opt.train_from,\n                                map_location=lambda storage, loc: storage)\n\n        model_opt = ArgumentParser.ckpt_model_opts(checkpoint[\"opt\"])\n        ArgumentParser.update_model_opts(model_opt)\n        ArgumentParser.validate_model_opts(model_opt)\n        logger.info('Loading vocab from checkpoint at %s.' % opt.train_from)\n        vocab = checkpoint['vocab']\n    else:\n        checkpoint = None\n        model_opt = opt\n        vocab = torch.load(opt.data + '.vocab.pt')\n\n    # check for code where vocab is saved instead of fields\n    # (in the future this will be done in a smarter way)\n    if old_style_vocab(vocab):\n        fields = load_old_vocab(\n            vocab, opt.model_type, dynamic_dict=opt.copy_attn)\n    else:\n        fields = vocab\n\n    # Report src and tgt vocab sizes, including for features\n    for side in ['src', 'tgt']:\n        f = fields[side]\n        try:\n            f_iter = iter(f)\n        except TypeError:\n            f_iter = [(side, f)]\n        for sn, sf in f_iter:\n            if sf.use_vocab:\n                logger.info(' * %s vocab size = %d' % (sn, len(sf.vocab)))\n\n    # Build model.\n    model = build_model(model_opt, opt, fields, checkpoint)\n    n_params, enc, dec = _tally_parameters(model)\n    logger.info('encoder: %d' % enc)\n    logger.info('decoder: %d' % dec)\n    logger.info('* number of parameters: %d' % n_params)\n    _check_save_model_path(opt)\n\n    # Build optimizer.\n    optim = Optimizer.from_opt(model, opt, checkpoint=checkpoint)\n\n    # Build model saver\n    model_saver = build_model_saver(model_opt, opt, model, fields, optim)\n\n    trainer = build_trainer(\n        opt, device_id, model, fields, optim, model_saver=model_saver)\n\n    train_iterables = []\n    for train_id in opt.data_ids:\n        if train_id:\n            shard_base = \"train_\" + train_id\n        else:\n            shard_base = \"train\"\n        iterable = build_dataset_iter(shard_base, fields, opt, multi=True)\n        train_iterables.append(iterable)\n\n    train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)\n\n    valid_iter = build_dataset_iter(\n        \"valid\", fields, opt, is_train=False)\n\n    if len(opt.gpu_ranks):\n        logger.info('Starting training on GPU: %s' % opt.gpu_ranks)\n    else:\n        logger.info('Starting training on CPU, could be very slow')\n    train_steps = opt.train_steps\n    if opt.single_pass and train_steps > 0:\n        logger.warning(\"Option single_pass is enabled, ignoring train_steps.\")\n        train_steps = 0\n    trainer.train(\n        train_iter,\n        train_steps,\n        save_checkpoint_steps=opt.save_checkpoint_steps,\n        valid_iter=valid_iter,\n        valid_steps=opt.valid_steps)\n\n    if opt.tensorboard:\n        trainer.report_manager.tensorboard_writer.close()\n", "path": "onmt/train_single.py"}], "after_files": [{"content": "#!/usr/bin/env python\n\"\"\"Training on a single process.\"\"\"\nimport os\n\nimport torch\n\nfrom onmt.inputters.inputter import build_dataset_iter, \\\n    load_old_vocab, old_style_vocab, MultipleDatasetIterator\nfrom onmt.model_builder import build_model\nfrom onmt.utils.optimizers import Optimizer\nfrom onmt.utils.misc import set_random_seed\nfrom onmt.trainer import build_trainer\nfrom onmt.models import build_model_saver\nfrom onmt.utils.logging import init_logger, logger\nfrom onmt.utils.parse import ArgumentParser\n\n\ndef _check_save_model_path(opt):\n    save_model_path = os.path.abspath(opt.save_model)\n    model_dirname = os.path.dirname(save_model_path)\n    if not os.path.exists(model_dirname):\n        os.makedirs(model_dirname)\n\n\ndef _tally_parameters(model):\n    enc = 0\n    dec = 0\n    for name, param in model.named_parameters():\n        if 'encoder' in name:\n            enc += param.nelement()\n        else:\n            dec += param.nelement()\n    return enc + dec, enc, dec\n\n\ndef configure_process(opt, device_id):\n    if device_id >= 0:\n        torch.cuda.set_device(device_id)\n    set_random_seed(opt.seed, device_id >= 0)\n\n\ndef main(opt, device_id):\n    # NOTE: It's important that ``opt`` has been validated and updated\n    # at this point.\n    configure_process(opt, device_id)\n    init_logger(opt.log_file)\n    assert len(opt.accum_count) == len(opt.accum_steps), \\\n        'Number of accum_count values must match number of accum_steps'\n    # Load checkpoint if we resume from a previous training.\n    if opt.train_from:\n        logger.info('Loading checkpoint from %s' % opt.train_from)\n        checkpoint = torch.load(opt.train_from,\n                                map_location=lambda storage, loc: storage)\n\n        model_opt = ArgumentParser.ckpt_model_opts(checkpoint[\"opt\"])\n        ArgumentParser.update_model_opts(model_opt)\n        ArgumentParser.validate_model_opts(model_opt)\n        logger.info('Loading vocab from checkpoint at %s.' % opt.train_from)\n        vocab = checkpoint['vocab']\n    else:\n        checkpoint = None\n        model_opt = opt\n        vocab = torch.load(opt.data + '.vocab.pt')\n\n    # check for code where vocab is saved instead of fields\n    # (in the future this will be done in a smarter way)\n    if old_style_vocab(vocab):\n        fields = load_old_vocab(\n            vocab, opt.model_type, dynamic_dict=opt.copy_attn)\n    else:\n        fields = vocab\n\n    # Report src and tgt vocab sizes, including for features\n    for side in ['src', 'tgt']:\n        f = fields[side]\n        try:\n            f_iter = iter(f)\n        except TypeError:\n            f_iter = [(side, f)]\n        for sn, sf in f_iter:\n            if sf.use_vocab:\n                logger.info(' * %s vocab size = %d' % (sn, len(sf.vocab)))\n\n    # Build model.\n    model = build_model(model_opt, opt, fields, checkpoint)\n    n_params, enc, dec = _tally_parameters(model)\n    logger.info('encoder: %d' % enc)\n    logger.info('decoder: %d' % dec)\n    logger.info('* number of parameters: %d' % n_params)\n    _check_save_model_path(opt)\n\n    # Build optimizer.\n    optim = Optimizer.from_opt(model, opt, checkpoint=checkpoint)\n\n    # Build model saver\n    model_saver = build_model_saver(model_opt, opt, model, fields, optim)\n\n    trainer = build_trainer(\n        opt, device_id, model, fields, optim, model_saver=model_saver)\n\n    train_iterables = []\n    if len(opt.data_ids) > 1:\n        for train_id in opt.data_ids:\n            shard_base = \"train_\" + train_id\n            iterable = build_dataset_iter(shard_base, fields, opt, multi=True)\n            train_iterables.append(iterable)\n        train_iter = MultipleDatasetIterator(train_iterables, device_id, opt)\n    else:\n        train_iter = build_dataset_iter(\"train\", fields, opt)\n\n    valid_iter = build_dataset_iter(\n        \"valid\", fields, opt, is_train=False)\n\n    if len(opt.gpu_ranks):\n        logger.info('Starting training on GPU: %s' % opt.gpu_ranks)\n    else:\n        logger.info('Starting training on CPU, could be very slow')\n    train_steps = opt.train_steps\n    if opt.single_pass and train_steps > 0:\n        logger.warning(\"Option single_pass is enabled, ignoring train_steps.\")\n        train_steps = 0\n    trainer.train(\n        train_iter,\n        train_steps,\n        save_checkpoint_steps=opt.save_checkpoint_steps,\n        valid_iter=valid_iter,\n        valid_steps=opt.valid_steps)\n\n    if opt.tensorboard:\n        trainer.report_manager.tensorboard_writer.close()\n", "path": "onmt/train_single.py"}]}