Collecting git+https://github.com/cheind/py-thin-plate-spline Cloning https://github.com/cheind/py-thin-plate-spline to /tmp/pip-req-build-lr87xsyc Resolved https://github.com/cheind/py-thin-plate-spline to commit f6995795397118b7d0ac01aecd3f39ffbfad9dee Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting absl-py==1.4.0 Downloading absl_py-1.4.0-py3-none-any.whl.metadata (2.3 kB) Collecting appdirs==1.4.4 Downloading appdirs-1.4.4-py2.py3-none-any.whl.metadata (9.0 kB) Collecting cachetools==5.3.1 Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB) Collecting certifi==2023.5.7 Downloading certifi-2023.5.7-py3-none-any.whl.metadata (2.2 kB) Collecting charset-normalizer==3.1.0 Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (30 kB) Collecting click==8.1.4 Downloading click-8.1.4-py3-none-any.whl.metadata (3.0 kB) Collecting cmake==3.26.4 Downloading cmake-3.26.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.7 kB) Collecting contourpy==1.1.0 Downloading contourpy-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.7 kB) Collecting cycler==0.11.0 Downloading cycler-0.11.0-py3-none-any.whl.metadata (785 bytes) Collecting debugpy==1.5.1 Downloading debugpy-1.5.1-py2.py3-none-any.whl.metadata (1.2 kB) Collecting decord==0.6.0 Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl.metadata (422 bytes) Collecting docker-pycreds==0.4.0 Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl.metadata (1.8 kB) Collecting filelock==3.12.2 Downloading filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB) Collecting fonttools==4.40.0 Downloading fonttools-4.40.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (148 kB) Collecting gdown Downloading gdown-5.2.0-py3-none-any.whl.metadata (5.8 kB) Collecting gitdb==4.0.10 Downloading gitdb-4.0.10-py3-none-any.whl.metadata (1.1 kB) Collecting gitpython==3.1.31 Downloading GitPython-3.1.31-py3-none-any.whl.metadata (1.3 kB) Collecting google-auth-oauthlib==1.0.0 Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl.metadata (2.7 kB) Collecting google-auth==2.22.0 Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB) Collecting grpcio==1.56.2 Downloading grpcio-1.56.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB) Collecting hickle Downloading hickle-5.0.3-py3-none-any.whl.metadata (22 kB) Collecting idna==3.4 Downloading idna-3.4-py3-none-any.whl.metadata (9.8 kB) Collecting imageio==2.31.5 Downloading imageio-2.31.5-py3-none-any.whl.metadata (4.6 kB) Collecting jinja2==3.1.2 Downloading Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB) Collecting joblib==1.2.0 Downloading joblib-1.2.0-py3-none-any.whl.metadata (5.3 kB) Collecting jupyter-core==5.3.1 Downloading jupyter_core-5.3.1-py3-none-any.whl.metadata (3.4 kB) Collecting kiwisolver==1.4.4 Downloading kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.3 kB) Collecting kornia==0.6.12 Downloading kornia-0.6.12-py2.py3-none-any.whl.metadata (12 kB) Collecting lazy-loader==0.3 Downloading lazy_loader-0.3-py3-none-any.whl.metadata (4.3 kB) Collecting lit==16.0.6 Downloading lit-16.0.6.tar.gz (153 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting lzstring Downloading lzstring-1.0.4.tar.gz (4.3 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting markdown==3.4.3 Downloading Markdown-3.4.3-py3-none-any.whl.metadata (6.7 kB) Collecting markupsafe==2.1.3 Downloading MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB) Collecting matplotlib==3.7.1 Downloading matplotlib-3.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB) Collecting mpmath==1.3.0 Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB) Collecting natsort==8.4.0 Downloading natsort-8.4.0-py3-none-any.whl.metadata (21 kB) Collecting networkx==3.1 Downloading networkx-3.1-py3-none-any.whl.metadata (5.3 kB) Collecting numpy==1.25.0 Downloading numpy-1.25.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB) Collecting nvidia-cublas-cu11==11.10.3.66 Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-cuda-cupti-cu11==11.7.101 Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-cuda-nvrtc-cu11==11.7.99 Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB) Collecting nvidia-cuda-runtime-cu11==11.7.99 Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-cudnn-cu11==8.5.0.96 Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-cufft-cu11==10.9.0.58 Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB) Collecting nvidia-curand-cu11==10.2.10.91 Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-cusolver-cu11==11.4.0.1 Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-cusparse-cu11==11.7.4.91 Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB) Collecting nvidia-nccl-cu11==2.14.3 Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB) Collecting nvidia-nvtx-cu11==11.7.91 Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB) Collecting oauthlib==3.2.2 Downloading oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB) Collecting opencv-python-headless Downloading opencv_python_headless-4.10.0.84-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB) Collecting opencv-python==4.7.0.72 Downloading opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB) Collecting pandas==2.0.3 Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB) Collecting pathtools==0.1.2 Downloading pathtools-0.1.2.tar.gz (11 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting pillow==9.5.0 Downloading Pillow-9.5.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (9.5 kB) Collecting pip==23.1.2 Downloading pip-23.1.2-py3-none-any.whl.metadata (4.1 kB) Collecting progressbar2 Downloading progressbar2-4.5.0-py3-none-any.whl.metadata (16 kB) Collecting protobuf==4.23.4 Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes) Collecting psutil==5.9.0 Downloading psutil-5.9.0.tar.gz (478 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting pyasn1-modules==0.3.0 Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl.metadata (3.6 kB) Collecting pyasn1==0.5.0 Downloading pyasn1-0.5.0-py2.py3-none-any.whl.metadata (8.5 kB) Collecting pycocotools==2.0.6 Downloading pycocotools-2.0.6.tar.gz (24 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting pyparsing==3.1.0 Downloading pyparsing-3.1.0-py3-none-any.whl.metadata (4.9 kB) Collecting pytz==2023.3 Downloading pytz-2023.3-py2.py3-none-any.whl.metadata (22 kB) Collecting pyyaml==6.0 Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) Collecting pyzmq==25.1.0 Downloading pyzmq-25.1.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (4.8 kB) Collecting requests-oauthlib==1.3.1 Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl.metadata (10 kB) Collecting requests==2.31.0 Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) Collecting rsa==4.9 Downloading rsa-4.9-py3-none-any.whl.metadata (4.2 kB) Collecting scikit-image==0.22.0 Downloading scikit_image-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB) Collecting scikit-learn==1.2.2 Downloading scikit_learn-1.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB) Collecting scipy==1.11.0 Downloading scipy-1.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (59 kB) Collecting sentry-sdk==1.27.1 Downloading sentry_sdk-1.27.1-py2.py3-none-any.whl.metadata (8.8 kB) Collecting setproctitle==1.3.2 Downloading setproctitle-1.3.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.8 kB) Collecting setuptools==67.8.0 Downloading setuptools-67.8.0-py3-none-any.whl.metadata (6.2 kB) Collecting smmap==5.0.0 Downloading smmap-5.0.0-py3-none-any.whl.metadata (4.2 kB) Collecting sympy==1.12 Downloading sympy-1.12-py3-none-any.whl.metadata (12 kB) Collecting tensorboard-data-server==0.7.1 Downloading tensorboard_data_server-0.7.1-py3-none-manylinux2014_x86_64.whl.metadata (1.1 kB) Collecting tensorboard==2.13.0 Downloading tensorboard-2.13.0-py3-none-any.whl.metadata (1.8 kB) Collecting tensorboardx==2.6.1 Downloading tensorboardX-2.6.1-py2.py3-none-any.whl.metadata (5.6 kB) Collecting threadpoolctl==3.1.0 Downloading threadpoolctl-3.1.0-py3-none-any.whl.metadata (9.2 kB) Collecting tifffile==2023.9.26 Downloading tifffile-2023.9.26-py3-none-any.whl.metadata (30 kB) Collecting torch==2.0.1 Downloading torch-2.0.1-cp311-cp311-manylinux1_x86_64.whl.metadata (24 kB) Collecting torchvision==0.15.2 Downloading torchvision-0.15.2-cp311-cp311-manylinux1_x86_64.whl.metadata (11 kB) Collecting tornado==6.1 Downloading tornado-6.1.tar.gz (497 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting tqdm==4.65.0 Downloading tqdm-4.65.0-py3-none-any.whl.metadata (56 kB) Collecting triton==2.0.0 Downloading triton-2.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.0 kB) Collecting tzdata==2023.3 Downloading tzdata-2023.3-py2.py3-none-any.whl.metadata (1.4 kB) Collecting urllib3==1.26.16 Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB) Collecting wandb==0.15.5 Downloading wandb-0.15.5-py3-none-any.whl.metadata (8.2 kB) Collecting werkzeug==2.3.6 Downloading Werkzeug-2.3.6-py3-none-any.whl.metadata (4.1 kB) Collecting wheel==0.38.4 Downloading wheel-0.38.4-py3-none-any.whl.metadata (2.1 kB) Collecting six>=1.4.0 (from docker-pycreds==0.4.0) Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) Collecting platformdirs>=2.5 (from jupyter-core==5.3.1) Downloading platformdirs-4.3.2-py3-none-any.whl.metadata (11 kB) Collecting traitlets>=5.3 (from jupyter-core==5.3.1) Downloading traitlets-5.14.3-py3-none-any.whl.metadata (10 kB) Collecting packaging (from kornia==0.6.12) Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB) Collecting python-dateutil>=2.7 (from matplotlib==3.7.1) Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) Collecting typing-extensions (from torch==2.0.1) Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) Collecting beautifulsoup4 (from gdown) Downloading beautifulsoup4-4.12.3-py3-none-any.whl.metadata (3.8 kB) Collecting h5py>=2.10.0 (from hickle) Downloading h5py-3.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB) Collecting future>=0.14.0 (from lzstring) Downloading future-1.0.0-py3-none-any.whl.metadata (4.0 kB) Collecting python-utils>=3.8.1 (from progressbar2) Downloading python_utils-3.8.2-py2.py3-none-any.whl.metadata (9.7 kB) Collecting soupsieve>1.2 (from beautifulsoup4->gdown) Downloading soupsieve-2.6-py3-none-any.whl.metadata (4.6 kB) Collecting PySocks!=1.5.7,>=1.5.6 (from requests[socks]->gdown) Downloading PySocks-1.7.1-py3-none-any.whl.metadata (13 kB) Downloading absl_py-1.4.0-py3-none-any.whl (126 kB) Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB) Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB) Downloading certifi-2023.5.7-py3-none-any.whl (156 kB) Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (197 kB) Downloading click-8.1.4-py3-none-any.whl (98 kB) Downloading cmake-3.26.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (24.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.0/24.0 MB 131.2 MB/s eta 0:00:00 Downloading contourpy-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB) Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB) Downloading debugpy-1.5.1-py2.py3-none-any.whl (4.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 96.8 MB/s eta 0:00:00 Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl (13.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.6/13.6 MB 193.3 MB/s eta 0:00:00 Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB) Downloading filelock-3.12.2-py3-none-any.whl (10 kB) Downloading fonttools-4.40.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 33.8 MB/s eta 0:00:00 Downloading gitdb-4.0.10-py3-none-any.whl (62 kB) Downloading GitPython-3.1.31-py3-none-any.whl (184 kB) Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB) Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB) Downloading grpcio-1.56.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 181.4 MB/s eta 0:00:00 Downloading idna-3.4-py3-none-any.whl (61 kB) Downloading imageio-2.31.5-py3-none-any.whl (313 kB) Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) Downloading joblib-1.2.0-py3-none-any.whl (297 kB) Downloading jupyter_core-5.3.1-py3-none-any.whl (93 kB) Downloading kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 90.4 MB/s eta 0:00:00 Downloading kornia-0.6.12-py2.py3-none-any.whl (653 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 653.4/653.4 kB 73.4 MB/s eta 0:00:00 Downloading lazy_loader-0.3-py3-none-any.whl (9.1 kB) Downloading Markdown-3.4.3-py3-none-any.whl (93 kB) Downloading MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB) Downloading matplotlib-3.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 177.9 MB/s eta 0:00:00 Downloading mpmath-1.3.0-py3-none-any.whl (536 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 101.5 MB/s eta 0:00:00 Downloading natsort-8.4.0-py3-none-any.whl (38 kB) Downloading networkx-3.1-py3-none-any.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 221.3 MB/s eta 0:00:00 Downloading numpy-1.25.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.6/17.6 MB 152.1 MB/s eta 0:00:00 Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 192.1 MB/s eta 0:00:00 Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 189.1 MB/s eta 0:00:00 Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 149.4 MB/s eta 0:00:00 Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 166.1 MB/s eta 0:00:00 Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 156.8 MB/s eta 0:00:00 Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl (168.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.4/168.4 MB 173.3 MB/s eta 0:00:00 Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.6/54.6 MB 176.2 MB/s eta 0:00:00 Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.6/102.6 MB 191.9 MB/s eta 0:00:00 Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 173.2/173.2 MB 186.8 MB/s eta 0:00:00 Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.1/177.1 MB 190.2 MB/s eta 0:00:00 Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB) Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB) Downloading opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.8/61.8 MB 152.7 MB/s eta 0:00:00 Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 194.8 MB/s eta 0:00:00 Downloading Pillow-9.5.0-cp311-cp311-manylinux_2_28_x86_64.whl (3.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 173.3 MB/s eta 0:00:00 Downloading pip-23.1.2-py3-none-any.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 200.3 MB/s eta 0:00:00 Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl (304 kB) Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB) Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB) Downloading pyparsing-3.1.0-py3-none-any.whl (102 kB) Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB) Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.9/757.9 kB 158.3 MB/s eta 0:00:00 Downloading pyzmq-25.1.0-cp311-cp311-manylinux_2_28_x86_64.whl (1.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 138.8 MB/s eta 0:00:00 Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB) Downloading requests-2.31.0-py3-none-any.whl (62 kB) Downloading rsa-4.9-py3-none-any.whl (34 kB) Downloading scikit_image-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/14.7 MB 129.8 MB/s eta 0:00:00 Downloading scikit_learn-1.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 188.8 MB/s eta 0:00:00 Downloading scipy-1.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.2/36.2 MB 38.2 MB/s eta 0:00:00 Downloading sentry_sdk-1.27.1-py2.py3-none-any.whl (211 kB) Downloading setproctitle-1.3.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB) Downloading setuptools-67.8.0-py3-none-any.whl (1.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 124.7 MB/s eta 0:00:00 Downloading smmap-5.0.0-py3-none-any.whl (24 kB) Downloading sympy-1.12-py3-none-any.whl (5.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 169.5 MB/s eta 0:00:00 Downloading tensorboard_data_server-0.7.1-py3-none-manylinux2014_x86_64.whl (6.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 153.4 MB/s eta 0:00:00 Downloading tensorboard-2.13.0-py3-none-any.whl (5.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 211.7 MB/s eta 0:00:00 Downloading tensorboardX-2.6.1-py2.py3-none-any.whl (101 kB) Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB) Downloading tifffile-2023.9.26-py3-none-any.whl (222 kB) Downloading torch-2.0.1-cp311-cp311-manylinux1_x86_64.whl (619.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 619.9/619.9 MB 116.6 MB/s eta 0:00:00 Downloading torchvision-0.15.2-cp311-cp311-manylinux1_x86_64.whl (6.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 140.7 MB/s eta 0:00:00 Downloading tqdm-4.65.0-py3-none-any.whl (77 kB) Downloading triton-2.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.3/63.3 MB 146.3 MB/s eta 0:00:00 Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB) Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB) Downloading wandb-0.15.5-py3-none-any.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 130.5 MB/s eta 0:00:00 Downloading Werkzeug-2.3.6-py3-none-any.whl (242 kB) Downloading wheel-0.38.4-py3-none-any.whl (36 kB) Downloading gdown-5.2.0-py3-none-any.whl (18 kB) Downloading hickle-5.0.3-py3-none-any.whl (107 kB) Downloading opencv_python_headless-4.10.0.84-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (49.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.9/49.9 MB 192.3 MB/s eta 0:00:00 Downloading progressbar2-4.5.0-py3-none-any.whl (57 kB) Downloading future-1.0.0-py3-none-any.whl (491 kB) Downloading h5py-3.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 209.9 MB/s eta 0:00:00 Downloading packaging-24.1-py3-none-any.whl (53 kB) Downloading platformdirs-4.3.2-py3-none-any.whl (18 kB) Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Downloading python_utils-3.8.2-py2.py3-none-any.whl (27 kB) Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Downloading traitlets-5.14.3-py3-none-any.whl (85 kB) Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) Downloading beautifulsoup4-4.12.3-py3-none-any.whl (147 kB) Downloading PySocks-1.7.1-py3-none-any.whl (16 kB) Downloading soupsieve-2.6-py3-none-any.whl (36 kB) Building wheels for collected packages: lit, pathtools, psutil, pycocotools, tornado, thinplate, lzstring Building wheel for lit (pyproject.toml): started Building wheel for lit (pyproject.toml): finished with status 'done' Created wheel for lit: filename=lit-16.0.6-py3-none-any.whl size=93587 sha256=516307eb365c7a34c36ad29221b78af5c0ef46efb81fd376ca67440e71d8cb97 Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/ab/84/e4/5af8c76af9e5bee472e825f1451c18bb3b261d80a7b3ec7f8a Building wheel for pathtools (setup.py): started Building wheel for pathtools (setup.py): finished with status 'done' Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=0326b7279d7b87442eed741583d4bc390bd0e03d78541d1e4fc3097353aaeaa1 Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/ea/b7/8b/84e94095ea418b9442f5abeba4ca7b0ad52d3fe7b69d6238a6 Building wheel for psutil (setup.py): started Building wheel for psutil (setup.py): finished with status 'done' Created wheel for psutil: filename=psutil-5.9.0-cp311-cp311-linux_x86_64.whl size=230265 sha256=4baf93c0dfeed31b573e43dbfbfb59ef540a2e421cc8a92f3aeadb5e2aeb2711 Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/b7/3f/a5/6da60cf045cc7d875f3f6746c27e9afa159642fc5acaa5ae08 Building wheel for pycocotools (pyproject.toml): started Building wheel for pycocotools (pyproject.toml): finished with status 'done' Created wheel for pycocotools: filename=pycocotools-2.0.6-cp311-cp311-linux_x86_64.whl size=104817 sha256=be529d436a996e27c495e5386d7db3901cf44a266f3cb026b9a4e11c6d5b52d4 Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/ad/ca/ea/fb115e04c841c3f71fd369b7d9805a43a5193f4f9251bed0ec Building wheel for tornado (setup.py): started Building wheel for tornado (setup.py): finished with status 'done' Created wheel for tornado: filename=tornado-6.1-cp311-cp311-linux_x86_64.whl size=416972 sha256=f476e7f7a7d846a6eca8249c3363f64c2a5235972b195b59903afdd2b1d09a3f Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/f2/59/06/a9c85c7b17ec0fc9b1e2ae0c59e3d39255d5c0a38492e33fea Building wheel for thinplate (setup.py): started Building wheel for thinplate (setup.py): finished with status 'done' Created wheel for thinplate: filename=thinplate-1.0.0-py3-none-any.whl size=6699 sha256=71cbe9f9b9d8a98e0c3ee3d7acbd0b07fce9df2a2193e3a45d78b1139af842f0 Stored in directory: /tmp/pip-ephem-wheel-cache-78ai6_13/wheels/8d/a9/10/26074e234ee6057094ebcf080ff2f898ab16f418672fab12ad Building wheel for lzstring (setup.py): started Building wheel for lzstring (setup.py): finished with status 'done' Created wheel for lzstring: filename=lzstring-1.0.4-py2.py3-none-any.whl size=4568 sha256=6589aea31ca6756c59667ce84b666cf31546bb97732885657a3e4fa038594290 Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/a2/68/ae/ab2520f1ce001476a22df0405a1066359d6c4f14aad5f6aae1 Successfully built lit pathtools psutil pycocotools tornado thinplate lzstring Installing collected packages: pytz, pathtools, mpmath, lit, cmake, appdirs, wheel, urllib3, tzdata, typing-extensions, traitlets, tqdm, tornado, threadpoolctl, tensorboard-data-server, sympy, soupsieve, smmap, six, setuptools, setproctitle, pyzmq, pyyaml, PySocks, pyparsing, pyasn1, psutil, protobuf, platformdirs, pip, pillow, packaging, oauthlib, nvidia-nccl-cu11, nvidia-cufft-cu11, nvidia-cuda-nvrtc-cu11, numpy, networkx, natsort, markupsafe, markdown, lazy-loader, kiwisolver, joblib, idna, grpcio, future, fonttools, filelock, debugpy, cycler, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tifffile, tensorboardx, sentry-sdk, scipy, rsa, requests, python-utils, python-dateutil, pyasn1-modules, opencv-python-headless, opencv-python, nvidia-nvtx-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, lzstring, jupyter-core, jinja2, imageio, h5py, gitdb, docker-pycreds, decord, contourpy, beautifulsoup4, scikit-learn, scikit-image, requests-oauthlib, progressbar2, pandas, nvidia-cusolver-cu11, nvidia-cudnn-cu11, matplotlib, hickle, google-auth, gitpython, wandb, pycocotools, google-auth-oauthlib, gdown, tensorboard, triton, torch, torchvision, thinplate, kornia Attempting uninstall: wheel Found existing installation: wheel 0.44.0 Uninstalling wheel-0.44.0: Successfully uninstalled wheel-0.44.0 Attempting uninstall: setuptools Found existing installation: setuptools 73.0.1 Uninstalling setuptools-73.0.1: Successfully uninstalled setuptools-73.0.1 Attempting uninstall: pip Found existing installation: pip 24.2 Uninstalling pip-24.2: Successfully uninstalled pip-24.2 Successfully installed PySocks-1.7.1 absl-py-1.4.0 appdirs-1.4.4 beautifulsoup4-4.12.3 cachetools-5.3.1 certifi-2023.5.7 charset-normalizer-3.1.0 click-8.1.4 cmake-3.26.4 contourpy-1.1.0 cycler-0.11.0 debugpy-1.5.1 decord-0.6.0 docker-pycreds-0.4.0 filelock-3.12.2 fonttools-4.40.0 future-1.0.0 gdown-5.2.0 gitdb-4.0.10 gitpython-3.1.31 google-auth-2.22.0 google-auth-oauthlib-1.0.0 grpcio-1.56.2 h5py-3.11.0 hickle-5.0.3 idna-3.4 imageio-2.31.5 jinja2-3.1.2 joblib-1.2.0 jupyter-core-5.3.1 kiwisolver-1.4.4 kornia-0.6.12 lazy-loader-0.3 lit-16.0.6 lzstring-1.0.4 markdown-3.4.3 markupsafe-2.1.3 matplotlib-3.7.1 mpmath-1.3.0 natsort-8.4.0 networkx-3.1 numpy-1.25.0 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 oauthlib-3.2.2 opencv-python-4.7.0.72 opencv-python-headless-4.10.0.84 packaging-24.1 pandas-2.0.3 pathtools-0.1.2 pillow-9.5.0 pip-23.1.2 platformdirs-4.3.2 progressbar2-4.5.0 protobuf-4.23.4 psutil-5.9.0 pyasn1-0.5.0 pyasn1-modules-0.3.0 pycocotools-2.0.6 pyparsing-3.1.0 python-dateutil-2.9.0.post0 python-utils-3.8.2 pytz-2023.3 pyyaml-6.0 pyzmq-25.1.0 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 scikit-image-0.22.0 scikit-learn-1.2.2 scipy-1.11.0 sentry-sdk-1.27.1 setproctitle-1.3.2 setuptools-67.8.0 six-1.16.0 smmap-5.0.0 soupsieve-2.6 sympy-1.12 tensorboard-2.13.0 tensorboard-data-server-0.7.1 tensorboardx-2.6.1 thinplate-1.0.0 threadpoolctl-3.1.0 tifffile-2023.9.26 torch-2.0.1 torchvision-0.15.2 tornado-6.1 tqdm-4.65.0 traitlets-5.14.3 triton-2.0.0 typing-extensions-4.12.2 tzdata-2023.3 urllib3-1.26.16 wandb-0.15.5 werkzeug-2.3.6 wheel-0.38.4 CUDA Device count: 2 CUDA Device count: 2 CUDA Device count: 2 CUDA Device count: 2 I am rank 0 in this world of size 2! We are assuming 2 GPUs. We are now starting stage 3 I will take the role of logging! I am rank 1 in this world of size 2! We are assuming 2 GPUs. We are now starting stage 3 Single object mode: False I am rank 1 in this world of size 2! We are assuming 2 GPUs. We are now starting stage 3 Single object mode: False I am rank 0 in this world of size 2! We are assuming 2 GPUs. We are now starting stage 3 I will take the role of logging! git No Git Info hyperpara {'benchmark': False, 'no_amp': False, 'swap': False, 'enable_segswap': False, 'segswap_model': None, 'static_root': '../static', 'bl_root': '../BL30K', 'yv_root': '../YouTube', 'egoexo_root': '/data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap', 'davis_root': '../DAVIS', 'num_workers': 4, 'key_dim': 64, 'value_dim': 512, 'hidden_dim': 64, 'deep_update_prob': 0.2, 'stages': '3', 's0_batch_size': 16, 's0_iterations': 150000, 's0_finetune': 0, 's0_steps': [], 's0_lr': 1e-05, 's0_num_ref_frames': 2, 's0_num_frames': 3, 's0_start_warm': 20000, 's0_end_warm': 70000, 's1_batch_size': 8, 's1_iterations': 250000, 's1_finetune': 0, 's1_steps': [200000], 's1_lr': 1e-05, 's1_num_ref_frames': 3, 's1_num_frames': 8, 's1_start_warm': 20000, 's1_end_warm': 70000, 's2_batch_size': 8, 's2_iterations': 150000, 's2_finetune': 10000, 's2_steps': [120000], 's2_lr': 1e-05, 's2_num_ref_frames': 3, 's2_num_frames': 8, 's2_start_warm': 20000, 's2_end_warm': 70000, 's3_batch_size': 8, 's3_iterations': 100000, 's3_finetune': 10000, 's3_steps': [80000], 's3_lr': 1e-05, 's3_num_ref_frames': 3, 's3_num_frames': 4, 's3_start_warm': 20000, 's3_end_warm': 70000, 'gamma': 0.1, 'weight_decay': 0.05, 'load_network': '/data/work-gcp-europe-west4-a/yuqian_fu/Ego/models/XMem.pth', 'load_checkpoint': None, 'log_text_interval': 100, 'log_image_interval': 1000, 'save_network_interval': 1000, 'save_checkpoint_interval': 2000, 'exp_id': 'xmem_egoexo_tune_s3', 'debug': False, 'local_rank': 0, 'amp': True, 'batch_size': 4, 'iterations': 100000, 'finetune': 10000, 'steps': [80000], 'lr': 1e-05, 'num_ref_frames': 3, 'num_frames': 4, 'start_warm': 20000, 'end_warm': 70000, 'single_object': False, 'num_gpus': 2} Single object mode: False git No Git Info hyperpara {'benchmark': False, 'no_amp': False, 'swap': False, 'enable_segswap': False, 'segswap_model': None, 'static_root': '../static', 'bl_root': '../BL30K', 'yv_root': '../YouTube', 'egoexo_root': '/data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap', 'davis_root': '../DAVIS', 'num_workers': 4, 'key_dim': 64, 'value_dim': 512, 'hidden_dim': 64, 'deep_update_prob': 0.2, 'stages': '3', 's0_batch_size': 16, 's0_iterations': 150000, 's0_finetune': 0, 's0_steps': [], 's0_lr': 1e-05, 's0_num_ref_frames': 2, 's0_num_frames': 3, 's0_start_warm': 20000, 's0_end_warm': 70000, 's1_batch_size': 8, 's1_iterations': 250000, 's1_finetune': 0, 's1_steps': [200000], 's1_lr': 1e-05, 's1_num_ref_frames': 3, 's1_num_frames': 8, 's1_start_warm': 20000, 's1_end_warm': 70000, 's2_batch_size': 8, 's2_iterations': 150000, 's2_finetune': 10000, 's2_steps': [120000], 's2_lr': 1e-05, 's2_num_ref_frames': 3, 's2_num_frames': 8, 's2_start_warm': 20000, 's2_end_warm': 70000, 's3_batch_size': 8, 's3_iterations': 100000, 's3_finetune': 10000, 's3_steps': [80000], 's3_lr': 1e-05, 's3_num_ref_frames': 3, 's3_num_frames': 4, 's3_start_warm': 20000, 's3_end_warm': 70000, 'gamma': 0.1, 'weight_decay': 0.05, 'load_network': '/data/work-gcp-europe-west4-a/yuqian_fu/Ego/models/XMem.pth', 'load_checkpoint': None, 'log_text_interval': 100, 'log_image_interval': 1000, 'save_network_interval': 1000, 'save_checkpoint_interval': 2000, 'exp_id': 'xmem_egoexo_tune_s3', 'debug': False, 'local_rank': 0, 'amp': True, 'batch_size': 4, 'iterations': 100000, 'finetune': 10000, 'steps': [80000], 'lr': 1e-05, 'num_ref_frames': 3, 'num_frames': 4, 'start_warm': 20000, 'end_warm': 70000, 'single_object': False, 'num_gpus': 2} Single object mode: False model_size 62185224 I am loading network from a disk, as listed in configuration I am loading network from a disk, as listed in configuration model_size 62185224 I am loading network from a disk, as listed in configuration I am loading network from a disk, as listed in configuration Network weight loaded from memory.Network weight loaded from memory. Network weight loaded from /data/work-gcp-europe-west4-a/yuqian_fu/Ego/models/XMem.pth Network weight loaded from /data/work-gcp-europe-west4-a/yuqian_fu/Ego/models/XMem.pth Network weight loaded from memory. Network weight loaded from /data/work-gcp-europe-west4-a/yuqian_fu/Ego/models/XMem.pth Network weight loaded from memory. Network weight loaded from /data/work-gcp-europe-west4-a/yuqian_fu/Ego/models/XMem.pth 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap.3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090EgoExo dataset size: 3090 Renewed with max_skip=5 Renewed with max_skip=5 EgoExo dataset size: 3090 Renewed with max_skip=5 EgoExo dataset size: 3090 Renewed with max_skip=5 We approximately use 260 epochs.We approximately use 260 epochs. We approximately use 260 epochs.We approximately use 260 epochs. The skip value will change approximately at the following iterations: [10000, 30000, 90000]The skip value will change approximately at the following iterations: [10000, 30000, 90000] The skip value will change approximately at the following iterations: [10000, 30000, 90000]The skip value will change approximately at the following iterations: [10000, 30000, 90000] Current epoch: 1 Current epoch: 1Current epoch: 1 Current epoch: 1 xmem_egoexo_tune_s3 - It 385 [TRAIN] [time ]: 26.326623 xmem_egoexo_tune_s3 - It 385 [TRAIN] [time ]: 26.327072 xmem_egoexo_tune_s3 - It 385 [TRAIN] [total_loss ]: 1.9630768 xmem_egoexo_tune_s3 - It 385 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 385 [TRAIN] [ce_loss_1 ]: 0.0207262 xmem_egoexo_tune_s3 - It 385 [TRAIN] [dice_loss_1 ]: 0.5997622 xmem_egoexo_tune_s3 - It 385 [TRAIN] [ce_loss_2 ]: 0.0239976 xmem_egoexo_tune_s3 - It 385 [TRAIN] [dice_loss_2 ]: 0.6391031 xmem_egoexo_tune_s3 - It 385 [TRAIN] [ce_loss_3 ]: 0.0246612 xmem_egoexo_tune_s3 - It 385 [TRAIN] [dice_loss_3 ]: 0.6548261 xmem_egoexo_tune_s3 - It 385 [TRAIN] [iou ]: 0.3001694 xmem_egoexo_tune_s3 - It 385 [TRAIN] [total_loss ]: 1.9271796 xmem_egoexo_tune_s3 - It 385 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 385 [TRAIN] [ce_loss_1 ]: 0.0202294 xmem_egoexo_tune_s3 - It 385 [TRAIN] [dice_loss_1 ]: 0.5880547 xmem_egoexo_tune_s3 - It 385 [TRAIN] [ce_loss_2 ]: 0.0235891 xmem_egoexo_tune_s3 - It 385 [TRAIN] [dice_loss_2 ]: 0.6265825 xmem_egoexo_tune_s3 - It 385 [TRAIN] [ce_loss_3 ]: 0.0244976 xmem_egoexo_tune_s3 - It 385 [TRAIN] [dice_loss_3 ]: 0.6442261 xmem_egoexo_tune_s3 - It 385 [TRAIN] [iou ]: 0.3108642 Current epoch: 2 Current epoch: 2 Current epoch: 2 Current epoch: 2 xmem_egoexo_tune_s3 - It 771 [TRAIN] [time ]: 17.366277 xmem_egoexo_tune_s3 - It 771 [TRAIN] [time ]: 17.366685 xmem_egoexo_tune_s3 - It 771 [TRAIN] [total_loss ]: 1.4374012 xmem_egoexo_tune_s3 - It 771 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 771 [TRAIN] [ce_loss_1 ]: 0.0130019 xmem_egoexo_tune_s3 - It 771 [TRAIN] [dice_loss_1 ]: 0.4321885 xmem_egoexo_tune_s3 - It 771 [TRAIN] [ce_loss_2 ]: 0.0153394 xmem_egoexo_tune_s3 - It 771 [TRAIN] [dice_loss_2 ]: 0.4708838 xmem_egoexo_tune_s3 - It 771 [TRAIN] [ce_loss_3 ]: 0.0167058 xmem_egoexo_tune_s3 - It 771 [TRAIN] [dice_loss_3 ]: 0.4892815 xmem_egoexo_tune_s3 - It 771 [TRAIN] [iou ]: 0.4583819 xmem_egoexo_tune_s3 - It 771 [TRAIN] [total_loss ]: 1.4412586 xmem_egoexo_tune_s3 - It 771 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 771 [TRAIN] [ce_loss_1 ]: 0.0129108 xmem_egoexo_tune_s3 - It 771 [TRAIN] [dice_loss_1 ]: 0.4336698 xmem_egoexo_tune_s3 - It 771 [TRAIN] [ce_loss_2 ]: 0.0150062 xmem_egoexo_tune_s3 - It 771 [TRAIN] [dice_loss_2 ]: 0.4728092 xmem_egoexo_tune_s3 - It 771 [TRAIN] [ce_loss_3 ]: 0.0162552 xmem_egoexo_tune_s3 - It 771 [TRAIN] [dice_loss_3 ]: 0.4906072 xmem_egoexo_tune_s3 - It 771 [TRAIN] [iou ]: 0.4570977 Current epoch: 3 Current epoch: 3 Current epoch: 3 Current epoch: 3 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_1000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_1000.pth. xmem_egoexo_tune_s3 - It 1157 [TRAIN] [time ]: 17.287787 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [time ]: 17.289621 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [total_loss ]: 1.3348642 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [ce_loss_1 ]: 0.0118151 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [dice_loss_1 ]: 0.4028752 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [ce_loss_2 ]: 0.0124762 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [dice_loss_2 ]: 0.4339893 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [ce_loss_3 ]: 0.0148608 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [dice_loss_3 ]: 0.4588474 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [iou ]: 0.4914688 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [total_loss ]: 1.3443501 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [ce_loss_1 ]: 0.0117923 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [dice_loss_1 ]: 0.4054139 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [ce_loss_2 ]: 0.0124050 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [dice_loss_2 ]: 0.4374599 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [ce_loss_3 ]: 0.0147124 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [dice_loss_3 ]: 0.4625663 xmem_egoexo_tune_s3 - It 1157 [TRAIN] [iou ]: 0.4891421 Current epoch: 4 Current epoch: 4 Current epoch: 4 Current epoch: 4 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [time ]: 17.336733 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [total_loss ]: 1.2890151 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [ce_loss_1 ]: 0.0118880 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [dice_loss_1 ]: 0.3835255 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [ce_loss_2 ]: 0.0127974 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [dice_loss_2 ]: 0.4164177 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [ce_loss_3 ]: 0.0194035 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [dice_loss_3 ]: 0.4449827 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [iou ]: 0.5070235 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [time ]: 17.339390 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [total_loss ]: 1.3309963 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [ce_loss_1 ]: 0.0118243 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [dice_loss_1 ]: 0.3959992 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [ce_loss_2 ]: 0.0134215 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [dice_loss_2 ]: 0.4298661 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [ce_loss_3 ]: 0.0200917 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [dice_loss_3 ]: 0.4597933 xmem_egoexo_tune_s3 - It 1543 [TRAIN] [iou ]: 0.4978615 Current epoch: 5 Current epoch: 5 Current epoch: 5 Current epoch: 5 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [time ]: 17.204149 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [total_loss ]: 1.2134232 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [ce_loss_1 ]: 0.0100513 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [dice_loss_1 ]: 0.3656548 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [ce_loss_2 ]: 0.0117032 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [dice_loss_2 ]: 0.3945764 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [ce_loss_3 ]: 0.0125156 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [dice_loss_3 ]: 0.4189217 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [iou ]: 0.5264138 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [time ]: 17.211286 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [total_loss ]: 1.1659077 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [ce_loss_1 ]: 0.0096625 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [dice_loss_1 ]: 0.3509466 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [ce_loss_2 ]: 0.0115409 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [dice_loss_2 ]: 0.3813270 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [ce_loss_3 ]: 0.0121315 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [dice_loss_3 ]: 0.4002988 xmem_egoexo_tune_s3 - It 1929 [TRAIN] [iou ]: 0.5411596 Current epoch: 6 Current epoch: 6 Current epoch: 6 Current epoch: 6 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_2000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_2000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_2000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_2000.pth. xmem_egoexo_tune_s3 - It 2315 [TRAIN] [time ]: 17.227245 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [total_loss ]: 1.1449062 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [ce_loss_1 ]: 0.0105613 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [dice_loss_1 ]: 0.3416202 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [ce_loss_2 ]: 0.0111439 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [dice_loss_2 ]: 0.3721945 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [ce_loss_3 ]: 0.0129656 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [dice_loss_3 ]: 0.3964205 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [iou ]: 0.5485376 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [time ]: 17.230772 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [total_loss ]: 1.1638792 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [ce_loss_1 ]: 0.0108808 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [dice_loss_1 ]: 0.3479303 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [ce_loss_2 ]: 0.0112283 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [dice_loss_2 ]: 0.3786448 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [ce_loss_3 ]: 0.0129068 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [dice_loss_3 ]: 0.4022880 xmem_egoexo_tune_s3 - It 2315 [TRAIN] [iou ]: 0.5420842 Current epoch: 7 Current epoch: 7 Current epoch: 7 Current epoch: 7 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [time ]: 17.145550 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [time ]: 17.144847 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [total_loss ]: 1.1077121 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [ce_loss_1 ]: 0.0097082 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [dice_loss_1 ]: 0.3342654 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [ce_loss_2 ]: 0.0108173 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [dice_loss_2 ]: 0.3612714 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [ce_loss_3 ]: 0.0114320 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [dice_loss_3 ]: 0.3802175 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [iou ]: 0.5592049 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [total_loss ]: 1.0899891 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [ce_loss_1 ]: 0.0096085 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [dice_loss_1 ]: 0.3278658 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [ce_loss_2 ]: 0.0105472 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [dice_loss_2 ]: 0.3538666 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [ce_loss_3 ]: 0.0112839 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [dice_loss_3 ]: 0.3768168 xmem_egoexo_tune_s3 - It 2701 [TRAIN] [iou ]: 0.5658308 Current epoch: 8 Current epoch: 8 Current epoch: 8 Current epoch: 8 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_3000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_3000.pth. xmem_egoexo_tune_s3 - It 3087 [TRAIN] [time ]: 17.110273 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [total_loss ]: 1.0893781 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [ce_loss_1 ]: 0.0088374 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [dice_loss_1 ]: 0.3230016 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [ce_loss_2 ]: 0.0107509 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [dice_loss_2 ]: 0.3609083 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [ce_loss_3 ]: 0.0114650 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [dice_loss_3 ]: 0.3744145 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [iou ]: 0.5650082 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [time ]: 17.110378 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [total_loss ]: 1.1071137 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [ce_loss_1 ]: 0.0091798 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [dice_loss_1 ]: 0.3276070 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [ce_loss_2 ]: 0.0112980 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [dice_loss_2 ]: 0.3671102 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [ce_loss_3 ]: 0.0119131 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [dice_loss_3 ]: 0.3800052 xmem_egoexo_tune_s3 - It 3087 [TRAIN] [iou ]: 0.5601372 Current epoch: 9 Current epoch: 9 Current epoch: 9 Current epoch: 9 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [time ]: 17.252347 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [total_loss ]: 1.0498688 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [ce_loss_1 ]: 0.0088701 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [dice_loss_1 ]: 0.3123125 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [ce_loss_2 ]: 0.0097084 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [dice_loss_2 ]: 0.3419464 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [ce_loss_3 ]: 0.0115497 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [dice_loss_3 ]: 0.3654813 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [iou ]: 0.5789119 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [time ]: 17.252557 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [total_loss ]: 1.0376553 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [ce_loss_1 ]: 0.0089046 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [dice_loss_1 ]: 0.3102215 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [ce_loss_2 ]: 0.0094815 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [dice_loss_2 ]: 0.3370008 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [ce_loss_3 ]: 0.0110340 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [dice_loss_3 ]: 0.3610128 xmem_egoexo_tune_s3 - It 3473 [TRAIN] [iou ]: 0.5831646 Current epoch: 10 Current epoch: 10 Current epoch: 10 Current epoch: 10 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [time ]: 17.228024 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [time ]: 17.228022 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [total_loss ]: 1.0244812 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [ce_loss_1 ]: 0.0076633 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [dice_loss_1 ]: 0.2986138 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [ce_loss_2 ]: 0.0150291 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [dice_loss_2 ]: 0.3366727 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [ce_loss_3 ]: 0.0102654 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [dice_loss_3 ]: 0.3562366 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [iou ]: 0.5877245 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [total_loss ]: 1.0189120 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [ce_loss_1 ]: 0.0078511 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [dice_loss_1 ]: 0.2971227 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [ce_loss_2 ]: 0.0146185 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [dice_loss_2 ]: 0.3348797 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [ce_loss_3 ]: 0.0105526 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [dice_loss_3 ]: 0.3538872 xmem_egoexo_tune_s3 - It 3859 [TRAIN] [iou ]: 0.5899406 Current epoch: 11 Current epoch: 11 Current epoch: 11 Current epoch: 11 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_4000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_4000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_4000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_4000.pth. xmem_egoexo_tune_s3 - It 4245 [TRAIN] [time ]: 17.444423 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [total_loss ]: 1.0209610 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [ce_loss_1 ]: 0.0082090 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [dice_loss_1 ]: 0.3054210 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [ce_loss_2 ]: 0.0096977 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [dice_loss_2 ]: 0.3346187 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [ce_loss_3 ]: 0.0104663 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [dice_loss_3 ]: 0.3525480 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [iou ]: 0.5883526 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [time ]: 17.447627 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [total_loss ]: 1.0277469 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [ce_loss_1 ]: 0.0081935 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [dice_loss_1 ]: 0.3080472 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [ce_loss_2 ]: 0.0098983 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [dice_loss_2 ]: 0.3349435 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [ce_loss_3 ]: 0.0106891 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [dice_loss_3 ]: 0.3559750 xmem_egoexo_tune_s3 - It 4245 [TRAIN] [iou ]: 0.5859799 Current epoch: 12 Current epoch: 12 Current epoch: 12 Current epoch: 12 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [time ]: 17.212758 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [time ]: 17.209659 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [total_loss ]: 1.0198221 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [total_loss ]: 1.0112140 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [ce_loss_1 ]: 0.0078050 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [ce_loss_1 ]: 0.0076612 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [dice_loss_1 ]: 0.3041178 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [dice_loss_1 ]: 0.3006271 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [ce_loss_2 ]: 0.0097256 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [ce_loss_2 ]: 0.0095511 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [dice_loss_2 ]: 0.3330118 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [dice_loss_2 ]: 0.3293591 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [ce_loss_3 ]: 0.0103395 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [ce_loss_3 ]: 0.0102189 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [dice_loss_3 ]: 0.3548221 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [dice_loss_3 ]: 0.3537962 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [iou ]: 0.5879908 xmem_egoexo_tune_s3 - It 4631 [TRAIN] [iou ]: 0.5903824 Current epoch: 13 Current epoch: 13 Current epoch: 13 Current epoch: 13 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_5000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_5000.pth. xmem_egoexo_tune_s3 - It 5017 [TRAIN] [time ]: 17.236133 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [time ]: 17.236131 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [total_loss ]: 0.9725446 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [total_loss ]: 0.9790159 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [ce_loss_1 ]: 0.0078427 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [ce_loss_1 ]: 0.0078013 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [dice_loss_1 ]: 0.2916321 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [dice_loss_1 ]: 0.2934574 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [ce_loss_2 ]: 0.0092198 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [ce_loss_2 ]: 0.0093610 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [dice_loss_2 ]: 0.3205496 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [dice_loss_2 ]: 0.3233856 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [ce_loss_3 ]: 0.0096537 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [ce_loss_3 ]: 0.0097087 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [dice_loss_3 ]: 0.3336465 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [dice_loss_3 ]: 0.3353017 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [iou ]: 0.6030849 xmem_egoexo_tune_s3 - It 5017 [TRAIN] [iou ]: 0.6010079 Current epoch: 14 Current epoch: 14 Current epoch: 14 Current epoch: 14 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [time ]: 17.332608 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [total_loss ]: 0.9919451 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [ce_loss_1 ]: 0.0075405 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [dice_loss_1 ]: 0.2944352 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [ce_loss_2 ]: 0.0087055 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [dice_loss_2 ]: 0.3211776 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [ce_loss_3 ]: 0.0147405 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [dice_loss_3 ]: 0.3453456 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [iou ]: 0.5987336 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [time ]: 17.333155 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [total_loss ]: 0.9791336 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [ce_loss_1 ]: 0.0074494 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [dice_loss_1 ]: 0.2902823 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [ce_loss_2 ]: 0.0084842 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [dice_loss_2 ]: 0.3167376 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [ce_loss_3 ]: 0.0152566 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [dice_loss_3 ]: 0.3409231 xmem_egoexo_tune_s3 - It 5403 [TRAIN] [iou ]: 0.6022546 Current epoch: 15 Current epoch: 15 Current epoch: 15 Current epoch: 15 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [time ]: 17.190541 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [time ]: 17.190270 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [total_loss ]: 0.9875407 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [ce_loss_1 ]: 0.0088650 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [dice_loss_1 ]: 0.2939115 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [ce_loss_2 ]: 0.0100317 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [dice_loss_2 ]: 0.3249244 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [ce_loss_3 ]: 0.0106102 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [dice_loss_3 ]: 0.3391976 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [iou ]: 0.5994688 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [total_loss ]: 0.9745652 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [ce_loss_1 ]: 0.0087583 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [dice_loss_1 ]: 0.2891781 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [ce_loss_2 ]: 0.0100356 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [dice_loss_2 ]: 0.3222396 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [ce_loss_3 ]: 0.0103378 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [dice_loss_3 ]: 0.3340156 xmem_egoexo_tune_s3 - It 5789 [TRAIN] [iou ]: 0.6039028 Current epoch: 16 Current epoch: 16 Current epoch: 16 Current epoch: 16 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_6000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_6000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_6000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_6000.pth. xmem_egoexo_tune_s3 - It 6175 [TRAIN] [time ]: 17.369582 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [total_loss ]: 0.9500146 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [ce_loss_1 ]: 0.0078877 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [dice_loss_1 ]: 0.2873166 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [ce_loss_2 ]: 0.0081164 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [dice_loss_2 ]: 0.3080097 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [ce_loss_3 ]: 0.0094818 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [dice_loss_3 ]: 0.3292023 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [iou ]: 0.6107876 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [time ]: 17.369395 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [total_loss ]: 0.9569437 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [ce_loss_1 ]: 0.0076051 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [dice_loss_1 ]: 0.2878702 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [ce_loss_2 ]: 0.0081069 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [dice_loss_2 ]: 0.3118155 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [ce_loss_3 ]: 0.0092721 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [dice_loss_3 ]: 0.3322737 xmem_egoexo_tune_s3 - It 6175 [TRAIN] [iou ]: 0.6087510 Current epoch: 17 Current epoch: 17 Current epoch: 17 Current epoch: 17 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [time ]: 17.351912 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [total_loss ]: 0.9523862 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [ce_loss_1 ]: 0.0076718 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [dice_loss_1 ]: 0.2852446 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [ce_loss_2 ]: 0.0089150 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [dice_loss_2 ]: 0.3128220 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [ce_loss_3 ]: 0.0095505 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [dice_loss_3 ]: 0.3281820 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [iou ]: 0.6118655 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [time ]: 17.352903 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [total_loss ]: 0.9521752 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [ce_loss_1 ]: 0.0076367 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [dice_loss_1 ]: 0.2870542 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [ce_loss_2 ]: 0.0088502 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [dice_loss_2 ]: 0.3129493 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [ce_loss_3 ]: 0.0092352 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [dice_loss_3 ]: 0.3264493 xmem_egoexo_tune_s3 - It 6561 [TRAIN] [iou ]: 0.6118615 Current epoch: 18 Current epoch: 18 Current epoch: 18 Current epoch: 18 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [time ]: 17.271393 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [total_loss ]: 0.9221727 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [ce_loss_1 ]: 0.0070969 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [dice_loss_1 ]: 0.2733548 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [ce_loss_2 ]: 0.0082470 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [dice_loss_2 ]: 0.3040221 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [ce_loss_3 ]: 0.0091545 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [dice_loss_3 ]: 0.3202972 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [iou ]: 0.6206735 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [time ]: 17.275092 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [total_loss ]: 0.9187072 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [ce_loss_1 ]: 0.0070232 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [dice_loss_1 ]: 0.2725345 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [ce_loss_2 ]: 0.0080588 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [dice_loss_2 ]: 0.3028546 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [ce_loss_3 ]: 0.0090213 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [dice_loss_3 ]: 0.3192144 xmem_egoexo_tune_s3 - It 6947 [TRAIN] [iou ]: 0.6212548 Current epoch: 19 Current epoch: 19 Current epoch: 19 Current epoch: 19 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_7000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_7000.pth. xmem_egoexo_tune_s3 - It 7333 [TRAIN] [time ]: 17.291757 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [total_loss ]: 0.9398555 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [ce_loss_1 ]: 0.0073838 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [dice_loss_1 ]: 0.2806870 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [ce_loss_2 ]: 0.0083207 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [dice_loss_2 ]: 0.3062314 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [time ]: 17.295014 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [ce_loss_3 ]: 0.0094011 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [dice_loss_3 ]: 0.3278312 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [iou ]: 0.6150705 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [total_loss ]: 0.9332901 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [ce_loss_1 ]: 0.0075171 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [dice_loss_1 ]: 0.2785396 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [ce_loss_2 ]: 0.0081513 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [dice_loss_2 ]: 0.3033610 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [ce_loss_3 ]: 0.0095471 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [dice_loss_3 ]: 0.3261737 xmem_egoexo_tune_s3 - It 7333 [TRAIN] [iou ]: 0.6177871 Current epoch: 20Current epoch: 20 Current epoch: 20 Current epoch: 20 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [time ]: 17.338561 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [time ]: 17.338123 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [total_loss ]: 0.9337726 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [ce_loss_1 ]: 0.0067985 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [dice_loss_1 ]: 0.2782503 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [ce_loss_2 ]: 0.0075532 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [dice_loss_2 ]: 0.3073752 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [ce_loss_3 ]: 0.0086785 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [dice_loss_3 ]: 0.3251167 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [iou ]: 0.6163989 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [total_loss ]: 0.9252947 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [ce_loss_1 ]: 0.0068075 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [dice_loss_1 ]: 0.2747072 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [ce_loss_2 ]: 0.0075102 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [dice_loss_2 ]: 0.3034705 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [ce_loss_3 ]: 0.0088450 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [dice_loss_3 ]: 0.3239540 xmem_egoexo_tune_s3 - It 7719 [TRAIN] [iou ]: 0.6197115 Current epoch: 21 Current epoch: 21 Current epoch: 21 Current epoch: 21 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_8000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_8000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_8000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_8000.pth. xmem_egoexo_tune_s3 - It 8105 [TRAIN] [time ]: 17.293388 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [time ]: 17.294382 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [total_loss ]: 0.9025317 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [ce_loss_1 ]: 0.0072689 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [dice_loss_1 ]: 0.2667175 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [ce_loss_2 ]: 0.0079754 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [dice_loss_2 ]: 0.2949270 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [ce_loss_3 ]: 0.0091379 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [dice_loss_3 ]: 0.3165048 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [iou ]: 0.6277425 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [total_loss ]: 0.8911731 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [ce_loss_1 ]: 0.0070024 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [dice_loss_1 ]: 0.2632348 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [ce_loss_2 ]: 0.0076277 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [dice_loss_2 ]: 0.2903712 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [ce_loss_3 ]: 0.0088465 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [dice_loss_3 ]: 0.3140904 xmem_egoexo_tune_s3 - It 8105 [TRAIN] [iou ]: 0.6315896 Current epoch: 22 Current epoch: 22 Current epoch: 22 Current epoch: 22 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [time ]: 17.366709 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [time ]: 17.365849 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [total_loss ]: 0.8835418 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [ce_loss_1 ]: 0.0068425 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [dice_loss_1 ]: 0.2662362 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [ce_loss_2 ]: 0.0077776 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [dice_loss_2 ]: 0.2858447 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [ce_loss_3 ]: 0.0085905 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [dice_loss_3 ]: 0.3082501 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [iou ]: 0.6323812 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [total_loss ]: 0.8946050 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [ce_loss_1 ]: 0.0070856 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [dice_loss_1 ]: 0.2696720 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [ce_loss_2 ]: 0.0081599 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [dice_loss_2 ]: 0.2905781 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [ce_loss_3 ]: 0.0089441 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [dice_loss_3 ]: 0.3101651 xmem_egoexo_tune_s3 - It 8491 [TRAIN] [iou ]: 0.6298400 Current epoch: 23 Current epoch: 23 Current epoch: 23 Current epoch: 23 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [time ]: 17.288112 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [time ]: 17.288003 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [total_loss ]: 0.9026874 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [total_loss ]: 0.9120978 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [ce_loss_1 ]: 0.0069208 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [ce_loss_1 ]: 0.0068392 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [dice_loss_1 ]: 0.2748566 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [dice_loss_1 ]: 0.2763745 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [ce_loss_2 ]: 0.0075416 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [ce_loss_2 ]: 0.0078342 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [dice_loss_2 ]: 0.2907021 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [dice_loss_2 ]: 0.2957014 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [ce_loss_3 ]: 0.0084969 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [ce_loss_3 ]: 0.0089048 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [dice_loss_3 ]: 0.3141690 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [dice_loss_3 ]: 0.3164434 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [iou ]: 0.6271451 xmem_egoexo_tune_s3 - It 8877 [TRAIN] [iou ]: 0.6230235 Current epoch: 24 Current epoch: 24 Current epoch: 24 Current epoch: 24 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_9000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_9000.pth. xmem_egoexo_tune_s3 - It 9263 [TRAIN] [time ]: 17.375791 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [total_loss ]: 0.8893340 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [ce_loss_1 ]: 0.0064838 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [dice_loss_1 ]: 0.2624540 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [ce_loss_2 ]: 0.0077439 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [dice_loss_2 ]: 0.2887411 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [ce_loss_3 ]: 0.0090447 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [dice_loss_3 ]: 0.3148663 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [iou ]: 0.6324592 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [time ]: 17.375872 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [total_loss ]: 0.8806600 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [ce_loss_1 ]: 0.0065300 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [dice_loss_1 ]: 0.2602019 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [ce_loss_2 ]: 0.0075648 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [dice_loss_2 ]: 0.2865942 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [ce_loss_3 ]: 0.0087087 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [dice_loss_3 ]: 0.3110603 xmem_egoexo_tune_s3 - It 9263 [TRAIN] [iou ]: 0.6349420 Current epoch: 25 Current epoch: 25 Current epoch: 25 Current epoch: 25 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [time ]: 17.327534 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [time ]: 17.329286 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [total_loss ]: 0.9022024 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [ce_loss_1 ]: 0.0126726 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [dice_loss_1 ]: 0.2675598 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [ce_loss_2 ]: 0.0085049 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [dice_loss_2 ]: 0.2902843 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [ce_loss_3 ]: 0.0096246 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [dice_loss_3 ]: 0.3135561 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [iou ]: 0.6308362 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [total_loss ]: 0.9116822 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [ce_loss_1 ]: 0.0141976 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [dice_loss_1 ]: 0.2684851 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [ce_loss_2 ]: 0.0089191 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [dice_loss_2 ]: 0.2945180 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [ce_loss_3 ]: 0.0098886 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [dice_loss_3 ]: 0.3156735 xmem_egoexo_tune_s3 - It 9649 [TRAIN] [iou ]: 0.6276664 Current epoch: 26 Current epoch: 26 Current epoch: 26 Current epoch: 26 Changing skip to cur_skip=10 Changing skip to cur_skip=10 Changing skip to cur_skip=10Changing skip to cur_skip=10 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090 Renewed with max_skip=10 EgoExo dataset size: 3090 Renewed with max_skip=10 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090 Renewed with max_skip=10 EgoExo dataset size: 3090 Renewed with max_skip=10 Current epoch: 27 Current epoch: 27 Current epoch: 27 Current epoch: 27 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_10000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_10000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_10000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_10000.pth. xmem_egoexo_tune_s3 - It 10385 [TRAIN] [time ]: 37.606128 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [time ]: 37.607799 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [total_loss ]: 0.8968966 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [total_loss ]: 0.9004717 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [ce_loss_1 ]: 0.0075924 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [ce_loss_1 ]: 0.0073729 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [dice_loss_1 ]: 0.2688172 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [dice_loss_1 ]: 0.2685364 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [ce_loss_2 ]: 0.0082116 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [ce_loss_2 ]: 0.0081231 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [dice_loss_2 ]: 0.2930835 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [dice_loss_2 ]: 0.2938465 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [ce_loss_3 ]: 0.0096118 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [ce_loss_3 ]: 0.0095025 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [dice_loss_3 ]: 0.3095797 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [dice_loss_3 ]: 0.3130901 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [iou ]: 0.6301156 xmem_egoexo_tune_s3 - It 10385 [TRAIN] [iou ]: 0.6286453 Current epoch: 28 Current epoch: 28 Current epoch: 28 Current epoch: 28 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [time ]: 17.332059 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [time ]: 17.333113 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [total_loss ]: 0.8909333 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [ce_loss_1 ]: 0.0066483 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [dice_loss_1 ]: 0.2692261 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [total_loss ]: 0.8855675 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [ce_loss_2 ]: 0.0079389 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [dice_loss_2 ]: 0.2860629 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [ce_loss_1 ]: 0.0062845 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [ce_loss_3 ]: 0.0088396 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [dice_loss_1 ]: 0.2669663 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [dice_loss_3 ]: 0.3122173 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [ce_loss_2 ]: 0.0076876 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [iou ]: 0.6317957 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [dice_loss_2 ]: 0.2850447 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [ce_loss_3 ]: 0.0085677 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [dice_loss_3 ]: 0.3110165 xmem_egoexo_tune_s3 - It 10771 [TRAIN] [iou ]: 0.6333277 Current epoch: 29 Current epoch: 29 Current epoch: 29 Current epoch: 29 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_11000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_11000.pth. xmem_egoexo_tune_s3 - It 11157 [TRAIN] [time ]: 17.338269 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [total_loss ]: 0.8926560 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [ce_loss_1 ]: 0.0070202 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [dice_loss_1 ]: 0.2672836 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [ce_loss_2 ]: 0.0075956 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [dice_loss_2 ]: 0.2908360 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [ce_loss_3 ]: 0.0085418 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [dice_loss_3 ]: 0.3113786 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [iou ]: 0.6315774 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [time ]: 17.338725 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [total_loss ]: 0.9025907 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [ce_loss_1 ]: 0.0071095 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [dice_loss_1 ]: 0.2691220 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [ce_loss_2 ]: 0.0077298 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [dice_loss_2 ]: 0.2932221 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [ce_loss_3 ]: 0.0088132 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [dice_loss_3 ]: 0.3165939 xmem_egoexo_tune_s3 - It 11157 [TRAIN] [iou ]: 0.6280782 Current epoch: 30 Current epoch: 30 Current epoch: 30 Current epoch: 30 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [time ]: 17.262498 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [total_loss ]: 0.8832973 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [ce_loss_1 ]: 0.0062756 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [time ]: 17.264037 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [dice_loss_1 ]: 0.2669585 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [total_loss ]: 0.8847644 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [ce_loss_2 ]: 0.0074671 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [dice_loss_2 ]: 0.2860944 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [ce_loss_1 ]: 0.0064912 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [ce_loss_3 ]: 0.0082449 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [dice_loss_1 ]: 0.2660557 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [dice_loss_3 ]: 0.3082567 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [ce_loss_2 ]: 0.0075159 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [iou ]: 0.6337507 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [dice_loss_2 ]: 0.2877334 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [ce_loss_3 ]: 0.0084380 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [dice_loss_3 ]: 0.3085299 xmem_egoexo_tune_s3 - It 11543 [TRAIN] [iou ]: 0.6330494 Current epoch: 31 Current epoch: 31 Current epoch: 31 Current epoch: 31 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [time ]: 17.410457 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [time ]: 17.411159 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [total_loss ]: 0.8920884 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [ce_loss_1 ]: 0.0069544 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [dice_loss_1 ]: 0.2640838 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [ce_loss_2 ]: 0.0080713 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [dice_loss_2 ]: 0.2870348 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [ce_loss_3 ]: 0.0089972 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [dice_loss_3 ]: 0.3169466 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [iou ]: 0.6314011 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [total_loss ]: 0.9182960 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [ce_loss_1 ]: 0.0077385 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [dice_loss_1 ]: 0.2709398 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [ce_loss_2 ]: 0.0086057 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [dice_loss_2 ]: 0.2952569 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [ce_loss_3 ]: 0.0094601 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [dice_loss_3 ]: 0.3262949 xmem_egoexo_tune_s3 - It 11929 [TRAIN] [iou ]: 0.6229287 Current epoch: 32 Current epoch: 32 Current epoch: 32 Current epoch: 32 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_12000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_12000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_12000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_12000.pth. xmem_egoexo_tune_s3 - It 12315 [TRAIN] [time ]: 17.551562 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [total_loss ]: 0.8985861 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [ce_loss_1 ]: 0.0127216 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [dice_loss_1 ]: 0.2670189 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [ce_loss_2 ]: 0.0078074 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [dice_loss_2 ]: 0.2907140 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [ce_loss_3 ]: 0.0092428 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [dice_loss_3 ]: 0.3110812 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [iou ]: 0.6314867 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [time ]: 17.554930 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [total_loss ]: 0.8912219 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [ce_loss_1 ]: 0.0126399 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [dice_loss_1 ]: 0.2651752 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [ce_loss_2 ]: 0.0077405 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [dice_loss_2 ]: 0.2871891 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [ce_loss_3 ]: 0.0088884 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [dice_loss_3 ]: 0.3095886 xmem_egoexo_tune_s3 - It 12315 [TRAIN] [iou ]: 0.6338956 Current epoch: 33 Current epoch: 33 Current epoch: 33 Current epoch: 33 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [time ]: 17.428569 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [total_loss ]: 0.8843774 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [ce_loss_1 ]: 0.0069388 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [dice_loss_1 ]: 0.2623208 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [ce_loss_2 ]: 0.0075816 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [dice_loss_2 ]: 0.2845015 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [ce_loss_3 ]: 0.0089643 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [dice_loss_3 ]: 0.3140702 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [iou ]: 0.6333131 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [time ]: 17.431356 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [total_loss ]: 0.9122688 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [ce_loss_1 ]: 0.0073122 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [dice_loss_1 ]: 0.2708262 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [ce_loss_2 ]: 0.0080022 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [dice_loss_2 ]: 0.2907660 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [ce_loss_3 ]: 0.0094269 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [dice_loss_3 ]: 0.3259351 xmem_egoexo_tune_s3 - It 12701 [TRAIN] [iou ]: 0.6240681 Current epoch: 34 Current epoch: 34 Current epoch: 34 Current epoch: 34 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_13000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_13000.pth. xmem_egoexo_tune_s3 - It 13087 [TRAIN] [time ]: 17.381110 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [time ]: 17.381239 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [total_loss ]: 0.8734003 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [ce_loss_1 ]: 0.0068424 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [dice_loss_1 ]: 0.2669465 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [ce_loss_2 ]: 0.0074295 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [dice_loss_2 ]: 0.2844884 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [ce_loss_3 ]: 0.0080797 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [dice_loss_3 ]: 0.2996137 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [iou ]: 0.6380248 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [total_loss ]: 0.8735018 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [ce_loss_1 ]: 0.0066665 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [dice_loss_1 ]: 0.2660036 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [ce_loss_2 ]: 0.0075407 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [dice_loss_2 ]: 0.2849388 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [ce_loss_3 ]: 0.0080456 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [dice_loss_3 ]: 0.3003064 xmem_egoexo_tune_s3 - It 13087 [TRAIN] [iou ]: 0.6376935 Current epoch: 35 Current epoch: 35 Current epoch: 35 Current epoch: 35 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [time ]: 17.380188 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [time ]: 17.380282 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [total_loss ]: 0.8632630 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [ce_loss_1 ]: 0.0067582 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [dice_loss_1 ]: 0.2560612 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [ce_loss_2 ]: 0.0076919 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [dice_loss_2 ]: 0.2808583 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [ce_loss_3 ]: 0.0085491 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [dice_loss_3 ]: 0.3033441 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [iou ]: 0.6409144 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [total_loss ]: 0.8679157 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [ce_loss_1 ]: 0.0068524 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [dice_loss_1 ]: 0.2556495 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [ce_loss_2 ]: 0.0076188 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [dice_loss_2 ]: 0.2825295 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [ce_loss_3 ]: 0.0083468 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [dice_loss_3 ]: 0.3069186 xmem_egoexo_tune_s3 - It 13473 [TRAIN] [iou ]: 0.6399687 Current epoch: 36 Current epoch: 36 Current epoch: 36 Current epoch: 36 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [time ]: 17.373065 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [total_loss ]: 0.8572934 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [ce_loss_1 ]: 0.0062161 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [dice_loss_1 ]: 0.2554022 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [ce_loss_2 ]: 0.0071576 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [dice_loss_2 ]: 0.2819637 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [ce_loss_3 ]: 0.0082561 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [dice_loss_3 ]: 0.2982974 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [iou ]: 0.6438627 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [time ]: 17.377288 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [total_loss ]: 0.8558124 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [ce_loss_1 ]: 0.0064624 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [dice_loss_1 ]: 0.2554910 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [ce_loss_2 ]: 0.0072938 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [dice_loss_2 ]: 0.2810275 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [ce_loss_3 ]: 0.0081051 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [dice_loss_3 ]: 0.2974323 xmem_egoexo_tune_s3 - It 13859 [TRAIN] [iou ]: 0.6442252 Current epoch: 37 Current epoch: 37 Current epoch: 37 Current epoch: 37 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_14000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_14000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_14000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_14000.pth. xmem_egoexo_tune_s3 - It 14245 [TRAIN] [time ]: 17.387759 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [time ]: 17.391952 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [total_loss ]: 0.8605376 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [ce_loss_1 ]: 0.0069811 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [dice_loss_1 ]: 0.2600437 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [ce_loss_2 ]: 0.0076526 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [dice_loss_2 ]: 0.2791029 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [ce_loss_3 ]: 0.0081993 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [dice_loss_3 ]: 0.2985578 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [iou ]: 0.6412630 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [total_loss ]: 0.8589408 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [ce_loss_1 ]: 0.0068052 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [dice_loss_1 ]: 0.2595748 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [ce_loss_2 ]: 0.0074011 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [dice_loss_2 ]: 0.2790147 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [ce_loss_3 ]: 0.0078318 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [dice_loss_3 ]: 0.2983131 xmem_egoexo_tune_s3 - It 14245 [TRAIN] [iou ]: 0.6423543 Current epoch: 38 Current epoch: 38 Current epoch: 38 Current epoch: 38 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [time ]: 17.505573 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [total_loss ]: 0.8550047 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [ce_loss_1 ]: 0.0065407 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [dice_loss_1 ]: 0.2592743 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [ce_loss_2 ]: 0.0072192 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [dice_loss_2 ]: 0.2746483 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [ce_loss_3 ]: 0.0086312 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [dice_loss_3 ]: 0.2986906 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [iou ]: 0.6422417 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [time ]: 17.505635 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [total_loss ]: 0.8510866 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [ce_loss_1 ]: 0.0065495 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [dice_loss_1 ]: 0.2581148 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [ce_loss_2 ]: 0.0068900 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [dice_loss_2 ]: 0.2736590 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [ce_loss_3 ]: 0.0083359 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [dice_loss_3 ]: 0.2975371 xmem_egoexo_tune_s3 - It 14631 [TRAIN] [iou ]: 0.6442621 Current epoch: 39 Current epoch: 39 Current epoch: 39 Current epoch: 39 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_15000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_15000.pth. xmem_egoexo_tune_s3 - It 15017 [TRAIN] [time ]: 17.392473 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [total_loss ]: 0.8403213 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [ce_loss_1 ]: 0.0122812 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [dice_loss_1 ]: 0.2467458 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [ce_loss_2 ]: 0.0074194 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [dice_loss_2 ]: 0.2715044 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [ce_loss_3 ]: 0.0083516 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [dice_loss_3 ]: 0.2940184 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [iou ]: 0.6500924 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [time ]: 17.395948 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [total_loss ]: 0.8498401 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [ce_loss_1 ]: 0.0123704 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [dice_loss_1 ]: 0.2505768 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [ce_loss_2 ]: 0.0075758 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [dice_loss_2 ]: 0.2751543 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [ce_loss_3 ]: 0.0084607 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [dice_loss_3 ]: 0.2957019 xmem_egoexo_tune_s3 - It 15017 [TRAIN] [iou ]: 0.6467585 Current epoch: 40 Current epoch: 40 Current epoch: 40 Current epoch: 40 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [time ]: 17.462990 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [total_loss ]: 0.8326903 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [ce_loss_1 ]: 0.0065377 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [dice_loss_1 ]: 0.2536860 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [ce_loss_2 ]: 0.0072689 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [dice_loss_2 ]: 0.2706836 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [ce_loss_3 ]: 0.0083825 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [dice_loss_3 ]: 0.2861313 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [iou ]: 0.6511020 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [time ]: 17.466975 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [total_loss ]: 0.8311821 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [ce_loss_1 ]: 0.0065873 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [dice_loss_1 ]: 0.2509352 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [ce_loss_2 ]: 0.0070231 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [dice_loss_2 ]: 0.2696689 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [ce_loss_3 ]: 0.0083325 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [dice_loss_3 ]: 0.2886348 xmem_egoexo_tune_s3 - It 15403 [TRAIN] [iou ]: 0.6520369 Current epoch: 41 Current epoch: 41 Current epoch: 41 Current epoch: 41 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [time ]: 17.342170 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [total_loss ]: 0.8344208 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [ce_loss_1 ]: 0.0060872 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [dice_loss_1 ]: 0.2524626 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [ce_loss_2 ]: 0.0068084 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [dice_loss_2 ]: 0.2707974 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [ce_loss_3 ]: 0.0075980 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [dice_loss_3 ]: 0.2906669 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [iou ]: 0.6495666 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [time ]: 17.343367 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [total_loss ]: 0.8341128 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [ce_loss_1 ]: 0.0061940 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [dice_loss_1 ]: 0.2517146 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [ce_loss_2 ]: 0.0069687 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [dice_loss_2 ]: 0.2717986 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [ce_loss_3 ]: 0.0077662 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [dice_loss_3 ]: 0.2896704 xmem_egoexo_tune_s3 - It 15789 [TRAIN] [iou ]: 0.6495385 Current epoch: 42 Current epoch: 42 Current epoch: 42 Current epoch: 42 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_16000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_16000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_16000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_16000.pth. xmem_egoexo_tune_s3 - It 16175 [TRAIN] [time ]: 17.384726 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [total_loss ]: 0.8507321 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [ce_loss_1 ]: 0.0067487 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [dice_loss_1 ]: 0.2535491 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [ce_loss_2 ]: 0.0125046 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [dice_loss_2 ]: 0.2745267 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [ce_loss_3 ]: 0.0082572 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [dice_loss_3 ]: 0.2951455 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [iou ]: 0.6466380 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [time ]: 17.384257 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [total_loss ]: 0.8446232 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [ce_loss_1 ]: 0.0067624 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [dice_loss_1 ]: 0.2507897 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [ce_loss_2 ]: 0.0122698 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [dice_loss_2 ]: 0.2736256 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [ce_loss_3 ]: 0.0079309 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [dice_loss_3 ]: 0.2932447 xmem_egoexo_tune_s3 - It 16175 [TRAIN] [iou ]: 0.6485030 Current epoch: 43 Current epoch: 43 Current epoch: 43 Current epoch: 43 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [time ]: 17.044893 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [time ]: 17.044896 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [total_loss ]: 0.8351644 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [ce_loss_1 ]: 0.0068107 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [dice_loss_1 ]: 0.2459699 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [ce_loss_2 ]: 0.0078508 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [dice_loss_2 ]: 0.2729964 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [ce_loss_3 ]: 0.0079226 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [dice_loss_3 ]: 0.2936137 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [iou ]: 0.6504556 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [total_loss ]: 0.8326444 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [ce_loss_1 ]: 0.0067791 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [dice_loss_1 ]: 0.2449742 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [ce_loss_2 ]: 0.0078945 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [dice_loss_2 ]: 0.2727874 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [ce_loss_3 ]: 0.0080288 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [dice_loss_3 ]: 0.2921802 xmem_egoexo_tune_s3 - It 16561 [TRAIN] [iou ]: 0.6511928 Current epoch: 44 Current epoch: 44 Current epoch: 44 Current epoch: 44 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [time ]: 16.981041 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [total_loss ]: 0.8291943 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [ce_loss_1 ]: 0.0063714 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [dice_loss_1 ]: 0.2492881 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [ce_loss_2 ]: 0.0076460 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [dice_loss_2 ]: 0.2677584 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [ce_loss_3 ]: 0.0082860 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [dice_loss_3 ]: 0.2898441 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [iou ]: 0.6518330 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [time ]: 16.983659 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [total_loss ]: 0.8270978 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [ce_loss_1 ]: 0.0064444 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [dice_loss_1 ]: 0.2481723 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [ce_loss_2 ]: 0.0075056 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [dice_loss_2 ]: 0.2679721 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [ce_loss_3 ]: 0.0081363 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [dice_loss_3 ]: 0.2888669 xmem_egoexo_tune_s3 - It 16947 [TRAIN] [iou ]: 0.6524211 Current epoch: 45 Current epoch: 45 Current epoch: 45 Current epoch: 45 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_17000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_17000.pth. xmem_egoexo_tune_s3 - It 17333 [TRAIN] [time ]: 16.886557 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [time ]: 16.889348 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [total_loss ]: 0.8156712 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [total_loss ]: 0.8147275 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [ce_loss_1 ]: 0.0063811 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [ce_loss_1 ]: 0.0063769 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [dice_loss_1 ]: 0.2488764 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [dice_loss_1 ]: 0.2482083 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [ce_loss_2 ]: 0.0069957 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [ce_loss_2 ]: 0.0072116 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [dice_loss_2 ]: 0.2627047 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [dice_loss_2 ]: 0.2611292 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [ce_loss_3 ]: 0.0077042 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [ce_loss_3 ]: 0.0081443 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [dice_loss_3 ]: 0.2830089 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [dice_loss_3 ]: 0.2836570 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [iou ]: 0.6568626 xmem_egoexo_tune_s3 - It 17333 [TRAIN] [iou ]: 0.6576206 Current epoch: 46 Current epoch: 46 Current epoch: 46 Current epoch: 46 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [time ]: 16.871353 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [time ]: 16.871421 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [total_loss ]: 0.8236466 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [ce_loss_1 ]: 0.0066831 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [dice_loss_1 ]: 0.2499457 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [ce_loss_2 ]: 0.0074611 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [dice_loss_2 ]: 0.2669204 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [ce_loss_3 ]: 0.0078014 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [dice_loss_3 ]: 0.2848347 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [iou ]: 0.6534544 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [total_loss ]: 0.8166371 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [ce_loss_1 ]: 0.0065673 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [dice_loss_1 ]: 0.2464687 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [ce_loss_2 ]: 0.0072480 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [dice_loss_2 ]: 0.2649348 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [ce_loss_3 ]: 0.0078146 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [dice_loss_3 ]: 0.2836035 xmem_egoexo_tune_s3 - It 17719 [TRAIN] [iou ]: 0.6555474 Current epoch: 47 Current epoch: 47 Current epoch: 47 Current epoch: 47 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_18000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_18000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_18000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_18000.pth. xmem_egoexo_tune_s3 - It 18105 [TRAIN] [time ]: 17.220408 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [time ]: 17.220765 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [total_loss ]: 0.8378596 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [ce_loss_1 ]: 0.0067535 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [dice_loss_1 ]: 0.2496168 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [ce_loss_2 ]: 0.0073217 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [dice_loss_2 ]: 0.2688065 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [ce_loss_3 ]: 0.0130253 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [dice_loss_3 ]: 0.2923357 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [iou ]: 0.6506015 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [total_loss ]: 0.8378151 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [ce_loss_1 ]: 0.0067258 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [dice_loss_1 ]: 0.2505472 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [ce_loss_2 ]: 0.0072380 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [dice_loss_2 ]: 0.2686536 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [ce_loss_3 ]: 0.0132604 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [dice_loss_3 ]: 0.2913897 xmem_egoexo_tune_s3 - It 18105 [TRAIN] [iou ]: 0.6498680 Current epoch: 48 Current epoch: 48 Current epoch: 48 Current epoch: 48 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [time ]: 16.984346 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [time ]: 16.984014 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [total_loss ]: 0.7987242 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [ce_loss_1 ]: 0.0065657 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [dice_loss_1 ]: 0.2441459 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [ce_loss_2 ]: 0.0068764 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [dice_loss_2 ]: 0.2577467 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [ce_loss_3 ]: 0.0075023 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [dice_loss_3 ]: 0.2758870 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [iou ]: 0.6619547 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [total_loss ]: 0.8002581 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [ce_loss_1 ]: 0.0062727 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [dice_loss_1 ]: 0.2438019 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [ce_loss_2 ]: 0.0069484 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [dice_loss_2 ]: 0.2587081 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [ce_loss_3 ]: 0.0076217 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [dice_loss_3 ]: 0.2769050 xmem_egoexo_tune_s3 - It 18491 [TRAIN] [iou ]: 0.6616296 Current epoch: 49Current epoch: 49 Current epoch: 49 Current epoch: 49 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [time ]: 16.990756 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [total_loss ]: 0.8132756 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [ce_loss_1 ]: 0.0064399 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [dice_loss_1 ]: 0.2420967 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [ce_loss_2 ]: 0.0068741 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [dice_loss_2 ]: 0.2613349 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [ce_loss_3 ]: 0.0130399 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [dice_loss_3 ]: 0.2834899 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [iou ]: 0.6585729 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [time ]: 16.993828 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [total_loss ]: 0.8065046 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [ce_loss_1 ]: 0.0062062 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [dice_loss_1 ]: 0.2409506 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [ce_loss_2 ]: 0.0068489 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [dice_loss_2 ]: 0.2581268 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [ce_loss_3 ]: 0.0134968 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [dice_loss_3 ]: 0.2808751 xmem_egoexo_tune_s3 - It 18877 [TRAIN] [iou ]: 0.6615135 Current epoch: 50 Current epoch: 50 Current epoch: 50 Current epoch: 50 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_19000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_19000.pth. xmem_egoexo_tune_s3 - It 19263 [TRAIN] [time ]: 16.855857 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [total_loss ]: 0.8071635 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [ce_loss_1 ]: 0.0061802 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [dice_loss_1 ]: 0.2418832 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [ce_loss_2 ]: 0.0120591 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [dice_loss_2 ]: 0.2608682 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [ce_loss_3 ]: 0.0074340 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [dice_loss_3 ]: 0.2787386 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [iou ]: 0.6610227 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [time ]: 16.859606 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [total_loss ]: 0.8089929 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [ce_loss_1 ]: 0.0059275 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [dice_loss_1 ]: 0.2410689 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [ce_loss_2 ]: 0.0120066 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [dice_loss_2 ]: 0.2630042 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [ce_loss_3 ]: 0.0074829 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [dice_loss_3 ]: 0.2795025 xmem_egoexo_tune_s3 - It 19263 [TRAIN] [iou ]: 0.6600888 Current epoch: 51 Current epoch: 51 Current epoch: 51 Current epoch: 51 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [time ]: 17.028869 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [time ]: 17.029547 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [total_loss ]: 0.8094299 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [total_loss ]: 0.7995327 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [p ]: 1.0000000 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [ce_loss_1 ]: 0.0063089 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [ce_loss_1 ]: 0.0063057 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [dice_loss_1 ]: 0.2464801 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [dice_loss_1 ]: 0.2436901 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [ce_loss_2 ]: 0.0069060 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [ce_loss_2 ]: 0.0069589 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [dice_loss_2 ]: 0.2625196 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [dice_loss_2 ]: 0.2600436 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [ce_loss_3 ]: 0.0075266 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [ce_loss_3 ]: 0.0077608 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [dice_loss_3 ]: 0.2796884 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [dice_loss_3 ]: 0.2747733 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [iou ]: 0.6585171 xmem_egoexo_tune_s3 - It 19649 [TRAIN] [iou ]: 0.6617586 Current epoch: 52 Current epoch: 52 Current epoch: 52 Current epoch: 52 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_20000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_20000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_20000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_20000.pth. xmem_egoexo_tune_s3 - It 20035 [TRAIN] [time ]: 17.059396 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [time ]: 17.059396 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [total_loss ]: 0.7849532 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [total_loss ]: 0.7928102 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [p ]: 0.9999722 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [p ]: 0.9999722 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [ce_loss_1 ]: 0.0109384 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [ce_loss_1 ]: 0.0111236 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [dice_loss_1 ]: 0.2331203 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [dice_loss_1 ]: 0.2361585 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [ce_loss_2 ]: 0.0068829 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [ce_loss_2 ]: 0.0070630 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [dice_loss_2 ]: 0.2546201 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [dice_loss_2 ]: 0.2562985 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [ce_loss_3 ]: 0.0070030 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [ce_loss_3 ]: 0.0072043 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [dice_loss_3 ]: 0.2723882 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [dice_loss_3 ]: 0.2749620 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [iou ]: 0.6671682 xmem_egoexo_tune_s3 - It 20035 [TRAIN] [iou ]: 0.6641426 Current epoch: 53 Current epoch: 53 Current epoch: 53 Current epoch: 53 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [time ]: 16.933678 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [total_loss ]: 0.7927576 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [p ]: 0.9961155 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [time ]: 16.934002 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [ce_loss_1 ]: 0.0056667 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [dice_loss_1 ]: 0.2359274 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [ce_loss_2 ]: 0.0114078 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [dice_loss_2 ]: 0.2539464 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [ce_loss_3 ]: 0.0068604 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [dice_loss_3 ]: 0.2789489 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [iou ]: 0.6659321 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [total_loss ]: 0.7879611 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [p ]: 0.9961155 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [ce_loss_1 ]: 0.0056933 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [dice_loss_1 ]: 0.2354602 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [ce_loss_2 ]: 0.0114280 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [dice_loss_2 ]: 0.2527546 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [ce_loss_3 ]: 0.0067787 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [dice_loss_3 ]: 0.2758458 xmem_egoexo_tune_s3 - It 20421 [TRAIN] [iou ]: 0.6671259 Current epoch: 54 Current epoch: 54 Current epoch: 54 Current epoch: 54 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [time ]: 16.955894 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [total_loss ]: 0.7715119 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [p ]: 0.9895535 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [ce_loss_1 ]: 0.0058580 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [dice_loss_1 ]: 0.2332636 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [ce_loss_2 ]: 0.0067686 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [dice_loss_2 ]: 0.2482347 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [ce_loss_3 ]: 0.0072223 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [dice_loss_3 ]: 0.2701644 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [iou ]: 0.6702619 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [time ]: 16.956345 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [total_loss ]: 0.7912991 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [p ]: 0.9895535 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [ce_loss_1 ]: 0.0061997 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [dice_loss_1 ]: 0.2408392 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [ce_loss_2 ]: 0.0069549 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [dice_loss_2 ]: 0.2558301 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [ce_loss_3 ]: 0.0074044 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [dice_loss_3 ]: 0.2740705 xmem_egoexo_tune_s3 - It 20807 [TRAIN] [iou ]: 0.6633548 Current epoch: 55 Current epoch: 55 Current epoch: 55 Current epoch: 55 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_21000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_21000.pth. xmem_egoexo_tune_s3 - It 21193 [TRAIN] [time ]: 17.069115 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [time ]: 17.070121 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [total_loss ]: 0.7811578 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [p ]: 0.9829915 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [ce_loss_1 ]: 0.0063082 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [dice_loss_1 ]: 0.2338694 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [ce_loss_2 ]: 0.0065945 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [dice_loss_2 ]: 0.2526202 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [ce_loss_3 ]: 0.0076206 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [dice_loss_3 ]: 0.2741446 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [iou ]: 0.6686309 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [total_loss ]: 0.7828432 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [p ]: 0.9829915 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [ce_loss_1 ]: 0.0061940 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [dice_loss_1 ]: 0.2343612 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [ce_loss_2 ]: 0.0068351 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [dice_loss_2 ]: 0.2523067 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [ce_loss_3 ]: 0.0078755 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [dice_loss_3 ]: 0.2752704 xmem_egoexo_tune_s3 - It 21193 [TRAIN] [iou ]: 0.6681686 Current epoch: 56 Current epoch: 56 Current epoch: 56 Current epoch: 56 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [time ]: 16.884066 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [total_loss ]: 0.7690761 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [p ]: 0.9764295 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [ce_loss_1 ]: 0.0062169 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [dice_loss_1 ]: 0.2310277 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [ce_loss_2 ]: 0.0072477 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [dice_loss_2 ]: 0.2474456 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [ce_loss_3 ]: 0.0076502 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [dice_loss_3 ]: 0.2694877 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [iou ]: 0.6727124 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [time ]: 16.883947 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [total_loss ]: 0.7774159 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [p ]: 0.9764295 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [ce_loss_1 ]: 0.0062290 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [dice_loss_1 ]: 0.2320011 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [ce_loss_2 ]: 0.0074546 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [dice_loss_2 ]: 0.2507418 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [ce_loss_3 ]: 0.0078640 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [dice_loss_3 ]: 0.2731252 xmem_egoexo_tune_s3 - It 21579 [TRAIN] [iou ]: 0.6701972 Current epoch: 57 Current epoch: 57 Current epoch: 57 Current epoch: 57 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [time ]: 16.833691 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [time ]: 16.833580 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [total_loss ]: 0.7795912 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [p ]: 0.9698675 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [ce_loss_1 ]: 0.0064473 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [dice_loss_1 ]: 0.2381542 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [ce_loss_2 ]: 0.0067417 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [dice_loss_2 ]: 0.2522888 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [ce_loss_3 ]: 0.0074158 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [dice_loss_3 ]: 0.2685431 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [iou ]: 0.6672483 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [total_loss ]: 0.7862516 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [p ]: 0.9698675 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [ce_loss_1 ]: 0.0064232 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [dice_loss_1 ]: 0.2387557 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [ce_loss_2 ]: 0.0066341 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [dice_loss_2 ]: 0.2541351 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [ce_loss_3 ]: 0.0076347 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [dice_loss_3 ]: 0.2726687 xmem_egoexo_tune_s3 - It 21965 [TRAIN] [iou ]: 0.6655315 Current epoch: 58 Current epoch: 58 Current epoch: 58 Current epoch: 58 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_22000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_22000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_22000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_22000.pth. xmem_egoexo_tune_s3 - It 22351 [TRAIN] [time ]: 16.850860 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [total_loss ]: 0.7661235 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [p ]: 0.9633054 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [ce_loss_1 ]: 0.0060732 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [dice_loss_1 ]: 0.2316508 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [time ]: 16.850896 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [ce_loss_2 ]: 0.0066037 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [total_loss ]: 0.7606599 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [dice_loss_2 ]: 0.2511224 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [p ]: 0.9633054 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [ce_loss_3 ]: 0.0075463 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [ce_loss_1 ]: 0.0059265 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [dice_loss_3 ]: 0.2631269 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [dice_loss_1 ]: 0.2303472 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [iou ]: 0.6721277 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [ce_loss_2 ]: 0.0066669 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [dice_loss_2 ]: 0.2481356 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [ce_loss_3 ]: 0.0073094 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [dice_loss_3 ]: 0.2622741 xmem_egoexo_tune_s3 - It 22351 [TRAIN] [iou ]: 0.6737014 Current epoch: 59 Current epoch: 59 Current epoch: 59 Current epoch: 59 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [time ]: 16.904457 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [total_loss ]: 0.7566398 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [p ]: 0.9567434 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [ce_loss_1 ]: 0.0059653 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [dice_loss_1 ]: 0.2271350 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [time ]: 16.905410 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [ce_loss_2 ]: 0.0060731 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [dice_loss_2 ]: 0.2439093 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [ce_loss_3 ]: 0.0072936 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [dice_loss_3 ]: 0.2662633 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [iou ]: 0.6751478 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [total_loss ]: 0.7557411 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [p ]: 0.9567434 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [ce_loss_1 ]: 0.0059866 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [dice_loss_1 ]: 0.2256823 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [ce_loss_2 ]: 0.0061928 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [dice_loss_2 ]: 0.2438903 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [ce_loss_3 ]: 0.0074230 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [dice_loss_3 ]: 0.2665659 xmem_egoexo_tune_s3 - It 22737 [TRAIN] [iou ]: 0.6760287 Current epoch: 60 Current epoch: 60 Current epoch: 60 Current epoch: 60 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_23000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_23000.pth. xmem_egoexo_tune_s3 - It 23123 [TRAIN] [time ]: 16.942766 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [total_loss ]: 0.7593171 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [p ]: 0.9501814 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [ce_loss_1 ]: 0.0057755 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [dice_loss_1 ]: 0.2302028 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [ce_loss_2 ]: 0.0064475 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [dice_loss_2 ]: 0.2440165 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [ce_loss_3 ]: 0.0074902 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [dice_loss_3 ]: 0.2653844 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [iou ]: 0.6742948 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [time ]: 16.942224 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [total_loss ]: 0.7534502 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [p ]: 0.9501814 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [ce_loss_1 ]: 0.0057074 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [dice_loss_1 ]: 0.2277640 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [ce_loss_2 ]: 0.0063321 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [dice_loss_2 ]: 0.2418989 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [ce_loss_3 ]: 0.0074343 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [dice_loss_3 ]: 0.2643132 xmem_egoexo_tune_s3 - It 23123 [TRAIN] [iou ]: 0.6775321 Current epoch: 61 Current epoch: 61 Current epoch: 61 Current epoch: 61 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [time ]: 16.852195 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [time ]: 16.852569 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [total_loss ]: 0.7645732 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [p ]: 0.9436194 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [ce_loss_1 ]: 0.0115897 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [dice_loss_1 ]: 0.2302781 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [ce_loss_2 ]: 0.0070134 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [dice_loss_2 ]: 0.2448334 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [ce_loss_3 ]: 0.0076209 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [dice_loss_3 ]: 0.2632375 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [iou ]: 0.6754506 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [total_loss ]: 0.7616592 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [p ]: 0.9436194 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [ce_loss_1 ]: 0.0115464 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [dice_loss_1 ]: 0.2291763 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [ce_loss_2 ]: 0.0068969 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [dice_loss_2 ]: 0.2439368 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [ce_loss_3 ]: 0.0074955 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [dice_loss_3 ]: 0.2626070 xmem_egoexo_tune_s3 - It 23509 [TRAIN] [iou ]: 0.6761837 Current epoch: 62 Current epoch: 62 Current epoch: 62 Current epoch: 62 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [time ]: 16.970785 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [total_loss ]: 0.7745357 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [p ]: 0.9370574 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [ce_loss_1 ]: 0.0063166 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [dice_loss_1 ]: 0.2349413 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [ce_loss_2 ]: 0.0070301 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [dice_loss_2 ]: 0.2507512 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [ce_loss_3 ]: 0.0073763 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [dice_loss_3 ]: 0.2681200 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [iou ]: 0.6693774 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [time ]: 16.973737 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [total_loss ]: 0.7580872 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [p ]: 0.9370574 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [ce_loss_1 ]: 0.0061296 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [dice_loss_1 ]: 0.2323501 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [ce_loss_2 ]: 0.0066621 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [dice_loss_2 ]: 0.2432634 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [ce_loss_3 ]: 0.0072889 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [dice_loss_3 ]: 0.2623928 xmem_egoexo_tune_s3 - It 23895 [TRAIN] [iou ]: 0.6748253 Current epoch: 63 Current epoch: 63 Current epoch: 63 Current epoch: 63 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_24000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_24000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_24000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_24000.pth. xmem_egoexo_tune_s3 - It 24281 [TRAIN] [time ]: 16.928785 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [time ]: 16.927549 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [total_loss ]: 0.7628237 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [p ]: 0.9304955 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [ce_loss_1 ]: 0.0060249 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [dice_loss_1 ]: 0.2306462 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [ce_loss_2 ]: 0.0069551 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [dice_loss_2 ]: 0.2479067 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [ce_loss_3 ]: 0.0073719 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [dice_loss_3 ]: 0.2639186 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [iou ]: 0.6743069 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [total_loss ]: 0.7657988 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [p ]: 0.9304955 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [ce_loss_1 ]: 0.0060303 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [dice_loss_1 ]: 0.2298003 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [ce_loss_2 ]: 0.0068656 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [dice_loss_2 ]: 0.2496981 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [ce_loss_3 ]: 0.0073500 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [dice_loss_3 ]: 0.2660542 xmem_egoexo_tune_s3 - It 24281 [TRAIN] [iou ]: 0.6730639 Current epoch: 64 Current epoch: 64 Current epoch: 64 Current epoch: 64 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [time ]: 16.806159 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [total_loss ]: 0.7428306 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [p ]: 0.9239335 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [ce_loss_1 ]: 0.0059108 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [dice_loss_1 ]: 0.2244736 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [ce_loss_2 ]: 0.0066578 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [dice_loss_2 ]: 0.2400009 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [ce_loss_3 ]: 0.0075489 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [dice_loss_3 ]: 0.2582384 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [iou ]: 0.6801959 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [time ]: 16.808001 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [total_loss ]: 0.7387712 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [p ]: 0.9239335 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [ce_loss_1 ]: 0.0061783 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [dice_loss_1 ]: 0.2236614 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [ce_loss_2 ]: 0.0067622 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [dice_loss_2 ]: 0.2379652 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [ce_loss_3 ]: 0.0077736 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [dice_loss_3 ]: 0.2564301 xmem_egoexo_tune_s3 - It 24667 [TRAIN] [iou ]: 0.6815653 Current epoch: 65 Current epoch: 65 Current epoch: 65 Current epoch: 65 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_25000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_25000.pth. xmem_egoexo_tune_s3 - It 25053 [TRAIN] [time ]: 16.896312 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [total_loss ]: 0.7656065 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [p ]: 0.9173715 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [ce_loss_1 ]: 0.0065365 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [time ]: 16.896469 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [dice_loss_1 ]: 0.2300922 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [ce_loss_2 ]: 0.0069247 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [total_loss ]: 0.7631714 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [dice_loss_2 ]: 0.2496877 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [p ]: 0.9173715 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [ce_loss_3 ]: 0.0133216 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [ce_loss_1 ]: 0.0066013 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [dice_loss_3 ]: 0.2590435 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [dice_loss_1 ]: 0.2305060 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [iou ]: 0.6761045 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [ce_loss_2 ]: 0.0071111 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [dice_loss_2 ]: 0.2473267 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [ce_loss_3 ]: 0.0138397 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [dice_loss_3 ]: 0.2577865 xmem_egoexo_tune_s3 - It 25053 [TRAIN] [iou ]: 0.6766639 Current epoch: 66 Current epoch: 66 Current epoch: 66 Current epoch: 66 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [time ]: 16.818547 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [time ]: 16.818726 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [total_loss ]: 0.7374533 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [p ]: 0.9108095 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [ce_loss_1 ]: 0.0063437 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [dice_loss_1 ]: 0.2220008 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [ce_loss_2 ]: 0.0070467 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [dice_loss_2 ]: 0.2388959 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [ce_loss_3 ]: 0.0077397 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [dice_loss_3 ]: 0.2554261 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [iou ]: 0.6821396 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [total_loss ]: 0.7379600 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [p ]: 0.9108095 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [ce_loss_1 ]: 0.0064504 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [dice_loss_1 ]: 0.2226556 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [ce_loss_2 ]: 0.0069225 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [dice_loss_2 ]: 0.2380406 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [ce_loss_3 ]: 0.0078005 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [dice_loss_3 ]: 0.2560902 xmem_egoexo_tune_s3 - It 25439 [TRAIN] [iou ]: 0.6822787 Current epoch: 67 Current epoch: 67 Current epoch: 67 Current epoch: 67 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [time ]: 16.958708 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [total_loss ]: 0.7728401 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [p ]: 0.9042475 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [ce_loss_1 ]: 0.0069302 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [dice_loss_1 ]: 0.2359781 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [ce_loss_2 ]: 0.0074527 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [dice_loss_2 ]: 0.2489103 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [ce_loss_3 ]: 0.0082568 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [dice_loss_3 ]: 0.2653118 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [iou ]: 0.6697497 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [time ]: 16.959360 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [total_loss ]: 0.7626206 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [p ]: 0.9042475 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [ce_loss_1 ]: 0.0069642 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [dice_loss_1 ]: 0.2338329 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [ce_loss_2 ]: 0.0072702 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [dice_loss_2 ]: 0.2449728 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [ce_loss_3 ]: 0.0078781 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [dice_loss_3 ]: 0.2617021 xmem_egoexo_tune_s3 - It 25825 [TRAIN] [iou ]: 0.6735110 Current epoch: 68 Current epoch: 68 Current epoch: 68 Current epoch: 68 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_26000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_26000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_26000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_26000.pth. xmem_egoexo_tune_s3 - It 26211 [TRAIN] [time ]: 16.920635 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [total_loss ]: 0.7464336 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [p ]: 0.8976855 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [ce_loss_1 ]: 0.0069429 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [dice_loss_1 ]: 0.2279829 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [ce_loss_2 ]: 0.0072193 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [dice_loss_2 ]: 0.2399151 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [ce_loss_3 ]: 0.0079147 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [dice_loss_3 ]: 0.2564584 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [iou ]: 0.6789016 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [time ]: 16.921716 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [total_loss ]: 0.7396110 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [p ]: 0.8976855 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [ce_loss_1 ]: 0.0066543 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [dice_loss_1 ]: 0.2264101 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [ce_loss_2 ]: 0.0071844 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [dice_loss_2 ]: 0.2368784 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [ce_loss_3 ]: 0.0079335 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [dice_loss_3 ]: 0.2545499 xmem_egoexo_tune_s3 - It 26211 [TRAIN] [iou ]: 0.6821120 Current epoch: 69 Current epoch: 69 Current epoch: 69 Current epoch: 69 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [time ]: 16.897996 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [total_loss ]: 0.7389112 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [p ]: 0.8911234 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [ce_loss_1 ]: 0.0061165 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [dice_loss_1 ]: 0.2243166 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [ce_loss_2 ]: 0.0066586 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [dice_loss_2 ]: 0.2411262 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [ce_loss_3 ]: 0.0077244 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [dice_loss_3 ]: 0.2529686 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [iou ]: 0.6825784 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [time ]: 16.899075 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [total_loss ]: 0.7455435 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [p ]: 0.8911234 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [ce_loss_1 ]: 0.0063390 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [dice_loss_1 ]: 0.2249234 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [ce_loss_2 ]: 0.0068298 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [dice_loss_2 ]: 0.2441990 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [ce_loss_3 ]: 0.0077721 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [dice_loss_3 ]: 0.2554800 xmem_egoexo_tune_s3 - It 26597 [TRAIN] [iou ]: 0.6804224 Current epoch: 70 Current epoch: 70 Current epoch: 70 Current epoch: 70 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [time ]: 16.842526 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [time ]: 16.844989 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [total_loss ]: 0.7540530 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [p ]: 0.8845614 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [ce_loss_1 ]: 0.0069732 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [dice_loss_1 ]: 0.2308949 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [ce_loss_2 ]: 0.0070168 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [dice_loss_2 ]: 0.2405101 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [ce_loss_3 ]: 0.0078191 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [dice_loss_3 ]: 0.2608387 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [iou ]: 0.6758179 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [total_loss ]: 0.7574735 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [p ]: 0.8845614 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [ce_loss_1 ]: 0.0069425 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [dice_loss_1 ]: 0.2303962 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [ce_loss_2 ]: 0.0071715 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [dice_loss_2 ]: 0.2424767 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [ce_loss_3 ]: 0.0079503 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [dice_loss_3 ]: 0.2625361 xmem_egoexo_tune_s3 - It 26983 [TRAIN] [iou ]: 0.6754094 Current epoch: 71 Current epoch: 71 Current epoch: 71 Current epoch: 71 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_27000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_27000.pth. xmem_egoexo_tune_s3 - It 27369 [TRAIN] [time ]: 17.081352 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [total_loss ]: 0.7305466 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [p ]: 0.8779994 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [ce_loss_1 ]: 0.0061732 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [dice_loss_1 ]: 0.2203456 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [ce_loss_2 ]: 0.0067867 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [dice_loss_2 ]: 0.2355684 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [ce_loss_3 ]: 0.0076096 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [dice_loss_3 ]: 0.2540628 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [iou ]: 0.6851359 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [time ]: 17.085025 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [total_loss ]: 0.7377119 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [p ]: 0.8779994 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [ce_loss_1 ]: 0.0063677 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [dice_loss_1 ]: 0.2236063 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [ce_loss_2 ]: 0.0068159 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [dice_loss_2 ]: 0.2379536 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [ce_loss_3 ]: 0.0075003 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [dice_loss_3 ]: 0.2554678 xmem_egoexo_tune_s3 - It 27369 [TRAIN] [iou ]: 0.6825662 Current epoch: 72 Current epoch: 72 Current epoch: 72 Current epoch: 72 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [time ]: 16.814409 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [time ]: 16.811547 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [total_loss ]: 0.7259706 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [p ]: 0.8714374 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [ce_loss_1 ]: 0.0066570 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [dice_loss_1 ]: 0.2219487 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [ce_loss_2 ]: 0.0064330 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [dice_loss_2 ]: 0.2359448 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [ce_loss_3 ]: 0.0066966 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [dice_loss_3 ]: 0.2482903 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [iou ]: 0.6863460 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [total_loss ]: 0.7325468 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [p ]: 0.8714374 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [ce_loss_1 ]: 0.0068521 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [dice_loss_1 ]: 0.2234779 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [ce_loss_2 ]: 0.0064180 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [dice_loss_2 ]: 0.2390369 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [ce_loss_3 ]: 0.0068021 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [dice_loss_3 ]: 0.2499595 xmem_egoexo_tune_s3 - It 27755 [TRAIN] [iou ]: 0.6846106 Current epoch: 73 Current epoch: 73 Current epoch: 73 Current epoch: 73 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_28000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_28000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_28000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_28000.pth. xmem_egoexo_tune_s3 - It 28141 [TRAIN] [time ]: 17.109271 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [total_loss ]: 0.7368962 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [p ]: 0.8648754 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [ce_loss_1 ]: 0.0114771 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [dice_loss_1 ]: 0.2203911 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [ce_loss_2 ]: 0.0067007 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [dice_loss_2 ]: 0.2331992 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [ce_loss_3 ]: 0.0078127 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [dice_loss_3 ]: 0.2573152 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [iou ]: 0.6848143 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [time ]: 17.112152 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [total_loss ]: 0.7415327 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [p ]: 0.8648754 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [ce_loss_1 ]: 0.0115729 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [dice_loss_1 ]: 0.2221260 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [ce_loss_2 ]: 0.0067121 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [dice_loss_2 ]: 0.2337847 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [ce_loss_3 ]: 0.0081684 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [dice_loss_3 ]: 0.2591684 xmem_egoexo_tune_s3 - It 28141 [TRAIN] [iou ]: 0.6832495 Current epoch: 74 Current epoch: 74 Current epoch: 74 Current epoch: 74 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [time ]: 16.900995 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [total_loss ]: 0.7280824 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [p ]: 0.8583135 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [ce_loss_1 ]: 0.0067516 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [dice_loss_1 ]: 0.2179532 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [ce_loss_2 ]: 0.0076983 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [dice_loss_2 ]: 0.2339553 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [ce_loss_3 ]: 0.0085826 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [dice_loss_3 ]: 0.2531412 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [iou ]: 0.6870473 Current epoch: 75 Current epoch: 75 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [time ]: 16.910204 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [total_loss ]: 0.7174422 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [p ]: 0.8583135 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [ce_loss_1 ]: 0.0065214 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [dice_loss_1 ]: 0.2140328 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [ce_loss_2 ]: 0.0075497 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [dice_loss_2 ]: 0.2308405 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [ce_loss_3 ]: 0.0079729 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [dice_loss_3 ]: 0.2505248 xmem_egoexo_tune_s3 - It 28527 [TRAIN] [iou ]: 0.6909179 Current epoch: 75 Current epoch: 75 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [time ]: 16.752555 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [total_loss ]: 0.7282465 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [p ]: 0.8517515 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [ce_loss_1 ]: 0.0065608 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [dice_loss_1 ]: 0.2179933 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [ce_loss_2 ]: 0.0072960 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [dice_loss_2 ]: 0.2361968 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [ce_loss_3 ]: 0.0080212 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [dice_loss_3 ]: 0.2521782 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [time ]: 16.740463 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [iou ]: 0.6860311 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [total_loss ]: 0.7295508 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [p ]: 0.8517515 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [ce_loss_1 ]: 0.0065604 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [dice_loss_1 ]: 0.2183952 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [ce_loss_2 ]: 0.0072505 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [dice_loss_2 ]: 0.2356391 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [ce_loss_3 ]: 0.0079770 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [dice_loss_3 ]: 0.2537285 xmem_egoexo_tune_s3 - It 28913 [TRAIN] [iou ]: 0.6864302 Current epoch: 76 Current epoch: 76 Current epoch: 76 Current epoch: 76 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_29000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_29000.pth. xmem_egoexo_tune_s3 - It 29299 [TRAIN] [time ]: 16.883767 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [total_loss ]: 0.7177404 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [p ]: 0.8451895 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [ce_loss_1 ]: 0.0067320 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [dice_loss_1 ]: 0.2160204 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [ce_loss_2 ]: 0.0069974 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [dice_loss_2 ]: 0.2312829 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [ce_loss_3 ]: 0.0074773 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [dice_loss_3 ]: 0.2492301 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [time ]: 16.883904 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [iou ]: 0.6899383 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [total_loss ]: 0.7179332 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [p ]: 0.8451895 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [ce_loss_1 ]: 0.0066769 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [dice_loss_1 ]: 0.2168870 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [ce_loss_2 ]: 0.0069594 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [dice_loss_2 ]: 0.2306177 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [ce_loss_3 ]: 0.0075843 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [dice_loss_3 ]: 0.2492077 xmem_egoexo_tune_s3 - It 29299 [TRAIN] [iou ]: 0.6909154 Current epoch: 77 Current epoch: 77 Current epoch: 77 Current epoch: 77 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [time ]: 17.076282 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [total_loss ]: 0.7205634 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [p ]: 0.8386275 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [ce_loss_1 ]: 0.0063977 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [dice_loss_1 ]: 0.2184614 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [ce_loss_2 ]: 0.0072168 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [dice_loss_2 ]: 0.2309952 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [ce_loss_3 ]: 0.0077749 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [dice_loss_3 ]: 0.2497171 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [iou ]: 0.6887178 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [time ]: 17.076308 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [total_loss ]: 0.7126493 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [p ]: 0.8386275 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [ce_loss_1 ]: 0.0064139 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [dice_loss_1 ]: 0.2159481 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [ce_loss_2 ]: 0.0073055 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [dice_loss_2 ]: 0.2290429 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [ce_loss_3 ]: 0.0077182 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [dice_loss_3 ]: 0.2462205 xmem_egoexo_tune_s3 - It 29685 [TRAIN] [iou ]: 0.6918876 Current epoch: 78 Current epoch: 78 Current epoch: 78 Current epoch: 78 Changing skip to cur_skip=15 Changing skip to cur_skip=15 Changing skip to cur_skip=15 Changing skip to cur_skip=15 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090 Renewed with max_skip=15 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090 Renewed with max_skip=15 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090 Renewed with max_skip=15 3090 out of 755 videos accepted in /data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap. EgoExo dataset size: 3090 Renewed with max_skip=15 Current epoch: 79 Current epoch: 79 Current epoch: 79 Current epoch: 79 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_30000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_30000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_30000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_30000.pth. xmem_egoexo_tune_s3 - It 30385 [TRAIN] [time ]: 34.591435 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [time ]: 34.591535 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [total_loss ]: 0.7203772 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [p ]: 0.8293964 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [ce_loss_1 ]: 0.0064424 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [dice_loss_1 ]: 0.2180129 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [ce_loss_2 ]: 0.0069214 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [dice_loss_2 ]: 0.2300248 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [ce_loss_3 ]: 0.0081353 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [dice_loss_3 ]: 0.2508402 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [iou ]: 0.6894856 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [total_loss ]: 0.7256650 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [p ]: 0.8293964 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [ce_loss_1 ]: 0.0065485 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [dice_loss_1 ]: 0.2188454 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [ce_loss_2 ]: 0.0069807 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [dice_loss_2 ]: 0.2312897 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [ce_loss_3 ]: 0.0082924 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [dice_loss_3 ]: 0.2537081 xmem_egoexo_tune_s3 - It 30385 [TRAIN] [iou ]: 0.6874651 Current epoch: 80 Current epoch: 80 Current epoch: 80 Current epoch: 80 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [time ]: 17.003964 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [time ]: 17.003968 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [total_loss ]: 0.7388233 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [p ]: 0.8201655 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [total_loss ]: 0.7405031 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [ce_loss_1 ]: 0.0070559 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [p ]: 0.8201655 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [dice_loss_1 ]: 0.2242802 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [ce_loss_1 ]: 0.0069919 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [ce_loss_2 ]: 0.0076611 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [dice_loss_1 ]: 0.2237696 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [dice_loss_2 ]: 0.2342321 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [ce_loss_2 ]: 0.0075826 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [ce_loss_3 ]: 0.0081660 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [dice_loss_2 ]: 0.2373372 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [dice_loss_3 ]: 0.2574277 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [ce_loss_3 ]: 0.0081471 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [iou ]: 0.6831134 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [dice_loss_3 ]: 0.2566745 xmem_egoexo_tune_s3 - It 30771 [TRAIN] [iou ]: 0.6825498 Current epoch: 81Current epoch: 81 Current epoch: 81 Current epoch: 81 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_31000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_31000.pth. xmem_egoexo_tune_s3 - It 31157 [TRAIN] [time ]: 16.932354 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [time ]: 16.932367 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [total_loss ]: 0.7370492 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [p ]: 0.8136035 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [total_loss ]: 0.7262853 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [p ]: 0.8136035 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [ce_loss_1 ]: 0.0063331 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [dice_loss_1 ]: 0.2172900 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [ce_loss_2 ]: 0.0074341 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [dice_loss_2 ]: 0.2343358 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [ce_loss_3 ]: 0.0079044 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [dice_loss_3 ]: 0.2529877 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [ce_loss_1 ]: 0.0067603 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [iou ]: 0.6873129 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [dice_loss_1 ]: 0.2198815 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [ce_loss_2 ]: 0.0077065 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [dice_loss_2 ]: 0.2361822 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [ce_loss_3 ]: 0.0082516 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [dice_loss_3 ]: 0.2582669 xmem_egoexo_tune_s3 - It 31157 [TRAIN] [iou ]: 0.6830811 Current epoch: 82 Current epoch: 82 Current epoch: 82 Current epoch: 82 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [time ]: 16.906514 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [total_loss ]: 0.7328574 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [p ]: 0.8070415 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [ce_loss_1 ]: 0.0070210 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [dice_loss_1 ]: 0.2191975 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [ce_loss_2 ]: 0.0072246 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [dice_loss_2 ]: 0.2396803 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [ce_loss_3 ]: 0.0079302 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [dice_loss_3 ]: 0.2518034 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [iou ]: 0.6847114 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [time ]: 16.909655 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [total_loss ]: 0.7253355 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [p ]: 0.8070415 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [ce_loss_1 ]: 0.0069149 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [dice_loss_1 ]: 0.2176194 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [ce_loss_2 ]: 0.0070192 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [dice_loss_2 ]: 0.2354178 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [ce_loss_3 ]: 0.0080274 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [dice_loss_3 ]: 0.2503367 xmem_egoexo_tune_s3 - It 31543 [TRAIN] [iou ]: 0.6864065 Current epoch: 83 Current epoch: 83 Current epoch: 83 Current epoch: 83 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [time ]: 16.820898 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [total_loss ]: 0.7292677 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [p ]: 0.8004794 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [ce_loss_1 ]: 0.0071211 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [dice_loss_1 ]: 0.2245860 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [ce_loss_2 ]: 0.0075658 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [dice_loss_2 ]: 0.2363723 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [ce_loss_3 ]: 0.0086829 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [dice_loss_3 ]: 0.2449393 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [iou ]: 0.6855429 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [time ]: 16.826633 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [total_loss ]: 0.7367951 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [p ]: 0.8004794 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [ce_loss_1 ]: 0.0074273 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [dice_loss_1 ]: 0.2262122 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [ce_loss_2 ]: 0.0078517 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [dice_loss_2 ]: 0.2367237 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [ce_loss_3 ]: 0.0090400 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [dice_loss_3 ]: 0.2495399 xmem_egoexo_tune_s3 - It 31929 [TRAIN] [iou ]: 0.6828955 Current epoch: 84 Current epoch: 84 Current epoch: 84 Current epoch: 84 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_32000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_32000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_32000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_32000.pth. xmem_egoexo_tune_s3 - It 32315 [TRAIN] [time ]: 17.253016 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [time ]: 17.255813 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [total_loss ]: 0.7452034 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [p ]: 0.7939174 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [ce_loss_1 ]: 0.0074952 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [dice_loss_1 ]: 0.2239264 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [ce_loss_2 ]: 0.0075536 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [dice_loss_2 ]: 0.2361655 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [ce_loss_3 ]: 0.0136193 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [dice_loss_3 ]: 0.2564431 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [iou ]: 0.6834129 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [total_loss ]: 0.7349876 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [p ]: 0.7939174 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [ce_loss_1 ]: 0.0072449 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [dice_loss_1 ]: 0.2217353 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [ce_loss_2 ]: 0.0073934 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [dice_loss_2 ]: 0.2329052 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [ce_loss_3 ]: 0.0135323 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [dice_loss_3 ]: 0.2521763 xmem_egoexo_tune_s3 - It 32315 [TRAIN] [iou ]: 0.6864967 Current epoch: 85 Current epoch: 85 Current epoch: 85 Current epoch: 85 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [time ]: 16.889503 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [total_loss ]: 0.7167245 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [p ]: 0.7873554 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [ce_loss_1 ]: 0.0071121 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [dice_loss_1 ]: 0.2164873 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [ce_loss_2 ]: 0.0073439 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [dice_loss_2 ]: 0.2286943 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [ce_loss_3 ]: 0.0078112 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [dice_loss_3 ]: 0.2492753 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [iou ]: 0.6903032 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [time ]: 16.892296 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [total_loss ]: 0.7134227 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [p ]: 0.7873554 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [ce_loss_1 ]: 0.0070659 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [dice_loss_1 ]: 0.2167871 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [ce_loss_2 ]: 0.0071261 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [dice_loss_2 ]: 0.2261749 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [ce_loss_3 ]: 0.0074379 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [dice_loss_3 ]: 0.2488306 xmem_egoexo_tune_s3 - It 32701 [TRAIN] [iou ]: 0.6912838 Current epoch: 86 Current epoch: 86 Current epoch: 86 Current epoch: 86 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_33000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_33000.pth. xmem_egoexo_tune_s3 - It 33087 [TRAIN] [time ]: 16.884986 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [total_loss ]: 0.7188795 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [p ]: 0.7807934 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [ce_loss_1 ]: 0.0067885 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [dice_loss_1 ]: 0.2202007 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [ce_loss_2 ]: 0.0071143 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [dice_loss_2 ]: 0.2259153 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [ce_loss_3 ]: 0.0134524 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [dice_loss_3 ]: 0.2454080 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [iou ]: 0.6919472 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [time ]: 16.885021 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [total_loss ]: 0.7144316 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [p ]: 0.7807934 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [ce_loss_1 ]: 0.0069258 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [dice_loss_1 ]: 0.2196897 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [ce_loss_2 ]: 0.0070720 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [dice_loss_2 ]: 0.2245064 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [ce_loss_3 ]: 0.0133706 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [dice_loss_3 ]: 0.2428668 xmem_egoexo_tune_s3 - It 33087 [TRAIN] [iou ]: 0.6929430 Current epoch: 87 Current epoch: 87 Current epoch: 87 Current epoch: 87 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [time ]: 16.879993 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [time ]: 16.883088 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [total_loss ]: 0.7198020 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [total_loss ]: 0.7195267 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [p ]: 0.7742314 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [p ]: 0.7742314 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [ce_loss_1 ]: 0.0068691 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [ce_loss_1 ]: 0.0070196 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [dice_loss_1 ]: 0.2176893 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [dice_loss_1 ]: 0.2178285 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [ce_loss_2 ]: 0.0074412 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [ce_loss_2 ]: 0.0075750 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [dice_loss_2 ]: 0.2289078 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [dice_loss_2 ]: 0.2301875 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [ce_loss_3 ]: 0.0082084 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [ce_loss_3 ]: 0.0079552 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [dice_loss_3 ]: 0.2506859 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [dice_loss_3 ]: 0.2489606 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [iou ]: 0.6902055 xmem_egoexo_tune_s3 - It 33473 [TRAIN] [iou ]: 0.6897718 Current epoch: 88 Current epoch: 88 Current epoch: 88 Current epoch: 88 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [time ]: 16.894239 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [time ]: 16.895670 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [total_loss ]: 0.7230765 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [p ]: 0.7676694 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [ce_loss_1 ]: 0.0066122 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [dice_loss_1 ]: 0.2204187 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [ce_loss_2 ]: 0.0072593 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [dice_loss_2 ]: 0.2297100 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [ce_loss_3 ]: 0.0083074 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [dice_loss_3 ]: 0.2507687 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [iou ]: 0.6885107 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [total_loss ]: 0.7248624 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [p ]: 0.7676694 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [ce_loss_1 ]: 0.0067556 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [dice_loss_1 ]: 0.2195470 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [ce_loss_2 ]: 0.0075319 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [dice_loss_2 ]: 0.2314421 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [ce_loss_3 ]: 0.0085812 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [dice_loss_3 ]: 0.2510045 xmem_egoexo_tune_s3 - It 33859 [TRAIN] [iou ]: 0.6878029 Current epoch: 89 Current epoch: 89 Current epoch: 89 Current epoch: 89 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_34000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_34000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_34000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_34000.pth. xmem_egoexo_tune_s3 - It 34245 [TRAIN] [time ]: 17.057199 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [time ]: 17.061018 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [total_loss ]: 0.7128356 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [p ]: 0.7611075 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [ce_loss_1 ]: 0.0071037 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [dice_loss_1 ]: 0.2167633 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [ce_loss_2 ]: 0.0076147 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [dice_loss_2 ]: 0.2274196 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [ce_loss_3 ]: 0.0081923 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [dice_loss_3 ]: 0.2457417 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [iou ]: 0.6928477 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [total_loss ]: 0.7139327 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [p ]: 0.7611075 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [ce_loss_1 ]: 0.0073022 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [dice_loss_1 ]: 0.2159749 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [ce_loss_2 ]: 0.0078741 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [dice_loss_2 ]: 0.2273064 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [ce_loss_3 ]: 0.0088552 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [dice_loss_3 ]: 0.2466197 xmem_egoexo_tune_s3 - It 34245 [TRAIN] [iou ]: 0.6927101 Current epoch: 90 Current epoch: 90 Current epoch: 90 Current epoch: 90 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [time ]: 17.081700 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [total_loss ]: 0.7170739 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [p ]: 0.7545455 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [ce_loss_1 ]: 0.0065594 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [dice_loss_1 ]: 0.2149927 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [ce_loss_2 ]: 0.0073524 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [dice_loss_2 ]: 0.2312512 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [ce_loss_3 ]: 0.0079484 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [dice_loss_3 ]: 0.2489694 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [time ]: 17.084083 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [iou ]: 0.6907858 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [total_loss ]: 0.7363997 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [p ]: 0.7545455 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [ce_loss_1 ]: 0.0070197 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [dice_loss_1 ]: 0.2186283 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [ce_loss_2 ]: 0.0081672 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [dice_loss_2 ]: 0.2372156 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [ce_loss_3 ]: 0.0089045 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [dice_loss_3 ]: 0.2564642 xmem_egoexo_tune_s3 - It 34631 [TRAIN] [iou ]: 0.6846171 Current epoch: 91 Current epoch: 91 Current epoch: 91 Current epoch: 91 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_35000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_35000.pth. xmem_egoexo_tune_s3 - It 35017 [TRAIN] [time ]: 16.896046 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [total_loss ]: 0.7184497 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [p ]: 0.7479835 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [ce_loss_1 ]: 0.0121178 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [dice_loss_1 ]: 0.2133018 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [ce_loss_2 ]: 0.0077874 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [dice_loss_2 ]: 0.2312983 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [ce_loss_3 ]: 0.0086422 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [dice_loss_3 ]: 0.2453018 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [iou ]: 0.6921067 Current epoch: 92 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [time ]: 16.899202 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [total_loss ]: 0.7204416 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [p ]: 0.7479835 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [ce_loss_1 ]: 0.0120012 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [dice_loss_1 ]: 0.2132704 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [ce_loss_2 ]: 0.0077901 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [dice_loss_2 ]: 0.2301416 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [ce_loss_3 ]: 0.0087225 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [dice_loss_3 ]: 0.2485154 xmem_egoexo_tune_s3 - It 35017 [TRAIN] [iou ]: 0.6906215 Current epoch: 92 Current epoch: 92 Current epoch: 92 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [time ]: 16.803997 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [time ]: 16.801140 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [total_loss ]: 0.7139157 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [p ]: 0.7414215 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [ce_loss_1 ]: 0.0071821 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [dice_loss_1 ]: 0.2131087 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [ce_loss_2 ]: 0.0077725 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [dice_loss_2 ]: 0.2290824 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [ce_loss_3 ]: 0.0083091 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [dice_loss_3 ]: 0.2484605 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [iou ]: 0.6915472 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [total_loss ]: 0.7033817 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [p ]: 0.7414215 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [ce_loss_1 ]: 0.0069035 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [dice_loss_1 ]: 0.2107927 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [ce_loss_2 ]: 0.0076894 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [dice_loss_2 ]: 0.2276278 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [ce_loss_3 ]: 0.0081372 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [dice_loss_3 ]: 0.2422308 xmem_egoexo_tune_s3 - It 35403 [TRAIN] [iou ]: 0.6959824 Current epoch: 93 Current epoch: 93 Current epoch: 93 Current epoch: 93 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [time ]: 16.931748 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [time ]: 16.931517 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [total_loss ]: 0.7195883 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [total_loss ]: 0.7187386 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [p ]: 0.7348595 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [p ]: 0.7348595 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [ce_loss_1 ]: 0.0077133 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [ce_loss_1 ]: 0.0073851 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [dice_loss_1 ]: 0.2165393 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [dice_loss_1 ]: 0.2164635 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [ce_loss_2 ]: 0.0084547 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [ce_loss_2 ]: 0.0080464 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [dice_loss_2 ]: 0.2299124 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [dice_loss_2 ]: 0.2302547 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [ce_loss_3 ]: 0.0088462 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [ce_loss_3 ]: 0.0085321 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [dice_loss_3 ]: 0.2481221 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [dice_loss_3 ]: 0.2480566 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [iou ]: 0.6914079 xmem_egoexo_tune_s3 - It 35789 [TRAIN] [iou ]: 0.6911691 Current epoch: 94 Current epoch: 94 Current epoch: 94 Current epoch: 94 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_36000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_36000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_36000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_36000.pth. xmem_egoexo_tune_s3 - It 36175 [TRAIN] [time ]: 17.178650 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [total_loss ]: 0.7070585 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [p ]: 0.7282974 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [ce_loss_1 ]: 0.0070096 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [dice_loss_1 ]: 0.2095819 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [ce_loss_2 ]: 0.0076734 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [dice_loss_2 ]: 0.2275309 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [ce_loss_3 ]: 0.0129356 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [dice_loss_3 ]: 0.2423269 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [iou ]: 0.6959402 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [time ]: 17.181632 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [total_loss ]: 0.7041671 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [p ]: 0.7282974 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [ce_loss_1 ]: 0.0067406 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [dice_loss_1 ]: 0.2086100 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [ce_loss_2 ]: 0.0078553 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [dice_loss_2 ]: 0.2274529 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [ce_loss_3 ]: 0.0130686 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [dice_loss_3 ]: 0.2404394 xmem_egoexo_tune_s3 - It 36175 [TRAIN] [iou ]: 0.6972770 Current epoch: 95 Current epoch: 95 Current epoch: 95 Current epoch: 95 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [time ]: 16.879543 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [total_loss ]: 0.6983845 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [p ]: 0.7217354 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [ce_loss_1 ]: 0.0066293 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [dice_loss_1 ]: 0.2018671 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [ce_loss_2 ]: 0.0076838 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [dice_loss_2 ]: 0.2280383 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [time ]: 16.877153 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [ce_loss_3 ]: 0.0082014 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [dice_loss_3 ]: 0.2459644 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [iou ]: 0.6968249 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [total_loss ]: 0.7048027 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [p ]: 0.7217354 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [ce_loss_1 ]: 0.0070354 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [dice_loss_1 ]: 0.2040018 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [ce_loss_2 ]: 0.0079187 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [dice_loss_2 ]: 0.2307261 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [ce_loss_3 ]: 0.0084236 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [dice_loss_3 ]: 0.2466970 xmem_egoexo_tune_s3 - It 36561 [TRAIN] [iou ]: 0.6946719 Current epoch: 96 Current epoch: 96 Current epoch: 96 Current epoch: 96 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [time ]: 16.895552 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [total_loss ]: 0.7063536 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [p ]: 0.7151734 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [ce_loss_1 ]: 0.0072472 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [dice_loss_1 ]: 0.2119061 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [ce_loss_2 ]: 0.0080302 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [dice_loss_2 ]: 0.2249939 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [ce_loss_3 ]: 0.0086273 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [dice_loss_3 ]: 0.2455487 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [iou ]: 0.6948809 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [time ]: 16.896545 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [total_loss ]: 0.7027585 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [p ]: 0.7151734 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [ce_loss_1 ]: 0.0072401 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [dice_loss_1 ]: 0.2112388 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [ce_loss_2 ]: 0.0077474 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [dice_loss_2 ]: 0.2253507 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [ce_loss_3 ]: 0.0082423 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [dice_loss_3 ]: 0.2429390 xmem_egoexo_tune_s3 - It 36947 [TRAIN] [iou ]: 0.6955306 Current epoch: 97 Current epoch: 97 Current epoch: 97 Current epoch: 97 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_37000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_37000.pth. xmem_egoexo_tune_s3 - It 37333 [TRAIN] [time ]: 16.859145 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [total_loss ]: 0.7189514 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [p ]: 0.7086114 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [ce_loss_1 ]: 0.0071597 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [dice_loss_1 ]: 0.2164450 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [ce_loss_2 ]: 0.0082170 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [dice_loss_2 ]: 0.2325320 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [ce_loss_3 ]: 0.0089160 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [dice_loss_3 ]: 0.2456814 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [iou ]: 0.6908400 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [time ]: 16.858824 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [total_loss ]: 0.7068181 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [p ]: 0.7086114 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [ce_loss_1 ]: 0.0072582 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [dice_loss_1 ]: 0.2145711 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [ce_loss_2 ]: 0.0078027 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [dice_loss_2 ]: 0.2298393 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [ce_loss_3 ]: 0.0082132 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [dice_loss_3 ]: 0.2391332 xmem_egoexo_tune_s3 - It 37333 [TRAIN] [iou ]: 0.6952108 Current epoch: 98 Current epoch: 98 Current epoch: 98 Current epoch: 98 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [time ]: 17.023203 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [total_loss ]: 0.6935734 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [p ]: 0.7020494 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [ce_loss_1 ]: 0.0069194 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [dice_loss_1 ]: 0.2082904 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [ce_loss_2 ]: 0.0074664 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [dice_loss_2 ]: 0.2235138 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [ce_loss_3 ]: 0.0086311 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [dice_loss_3 ]: 0.2387521 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [iou ]: 0.6988391 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [time ]: 17.023860 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [total_loss ]: 0.6951618 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [p ]: 0.7020494 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [ce_loss_1 ]: 0.0070391 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [dice_loss_1 ]: 0.2077032 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [ce_loss_2 ]: 0.0078862 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [dice_loss_2 ]: 0.2228186 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [ce_loss_3 ]: 0.0092261 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [dice_loss_3 ]: 0.2404884 xmem_egoexo_tune_s3 - It 37719 [TRAIN] [iou ]: 0.6995283 Current epoch: 99 Current epoch: 99 Current epoch: 99 Current epoch: 99 Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_38000.pth. Network saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_38000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_38000.pth. Checkpoint saved to /data/work-gcp-europe-west4-a/yuqian_fu/Ego/saves_xmem2/Sep10_20.14.00_xmem_egoexo_tune_s3/Sep10_20.14.00_xmem_egoexo_tune_s3_checkpoint_38000.pth. xmem_egoexo_tune_s3 - It 38105 [TRAIN] [time ]: 16.943846 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [time ]: 16.943384 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [total_loss ]: 0.6864870 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [p ]: 0.6954874 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [ce_loss_1 ]: 0.0068392 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [dice_loss_1 ]: 0.2078383 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [ce_loss_2 ]: 0.0074637 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [dice_loss_2 ]: 0.2210043 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [ce_loss_3 ]: 0.0076704 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [dice_loss_3 ]: 0.2356708 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [iou ]: 0.7008177 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [total_loss ]: 0.6934680 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [p ]: 0.6954874 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [ce_loss_1 ]: 0.0071880 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [dice_loss_1 ]: 0.2115061 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [ce_loss_2 ]: 0.0075660 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [dice_loss_2 ]: 0.2228371 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [ce_loss_3 ]: 0.0077206 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [dice_loss_3 ]: 0.2366499 xmem_egoexo_tune_s3 - It 38105 [TRAIN] [iou ]: 0.6984140 Current epoch: 100 Current epoch: 100 Current epoch: 100 Current epoch: 100 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [time ]: 17.081790 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [time ]: 17.081702 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [total_loss ]: 0.6833270 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [p ]: 0.6889255 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [ce_loss_1 ]: 0.0071634 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [dice_loss_1 ]: 0.2051358 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [ce_loss_2 ]: 0.0076076 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [dice_loss_2 ]: 0.2163749 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [ce_loss_3 ]: 0.0085877 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [dice_loss_3 ]: 0.2384575 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [iou ]: 0.7044566 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [total_loss ]: 0.6899236 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [p ]: 0.6889255 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [ce_loss_1 ]: 0.0073217 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [dice_loss_1 ]: 0.2065342 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [ce_loss_2 ]: 0.0075793 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [dice_loss_2 ]: 0.2177637 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [ce_loss_3 ]: 0.0086224 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [dice_loss_3 ]: 0.2421021 xmem_egoexo_tune_s3 - It 38491 [TRAIN] [iou ]: 0.7020434 Current epoch: 101 Current epoch: 101 Current epoch: 101 Current epoch: 101