<jupyter_start><jupyter_text>Caffe2 GPU Distributed  IntroductionThis example demonstrates how to run standard Caffe2 [resnet50_trainer.py](https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/resnet50_trainer.py) example using Batch AI. You can run it on a single or multiple compute nodes. Details- Standard Caffe2 sample script [resnet50_trainer.py](https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/resnet50_trainer.py) is used;- MNIST Dataset has been translated into a lmdb database, and can be obtained at http://download.caffe2.ai/databases/mnist-lmdb.zip;- Automatically created NFS folder will be used for rendezvous temp files to coordinate between each shard/node - Standard output of the job will be stored on Azure File Share.  Instructions Install Dependencies and Create Configuration file.Follow [instructions](/recipes) to install all dependencies and create configuration file.  Read Configuration and Create Batch AI client<jupyter_code>from __future__ import print_function

from datetime import datetime
import os
import sys
import zipfile

from azure.storage.file import FileService
from azure.storage.blob import BlockBlobService
import azure.mgmt.batchai.models as models

# The BatchAI/utilities folder contains helper functions used by different notebooks
sys.path.append('../../../')
import utilities as utils

cfg = utils.config.Configuration('../../configuration.json')
client = utils.config.create_batchai_client(cfg)<jupyter_output><empty_output><jupyter_text>Create Resoruce Group and Batch AI workspace if not exists:<jupyter_code>utils.config.create_resource_group(cfg)
_ = client.workspaces.create(cfg.resource_group, cfg.workspace, cfg.location).result()<jupyter_output><empty_output><jupyter_text>1. Prepare Training Dataset and Script in Azure Storage  Create Azure Blob ContainerWe will create a new Blob Container with name `batchaisample` under your storage account. This will be used to store the *input training dataset***Note** You don't need to create new blob Container for every cluster. We are doing this in this sample to simplify resource management for you.<jupyter_code>azure_blob_container_name = 'batchaisample'
blob_service = BlockBlobService(cfg.storage_account_name, cfg.storage_account_key)
blob_service.create_container(azure_blob_container_name, fail_on_exist=False)<jupyter_output><empty_output><jupyter_text>Upload MNIST Dataset to Azure Blob ContainerFor demonstration purposes, we will download preprocessed MNIST dataset to the current directory and upload it to Azure Blob Container directory named `mnist_dataset`.There are multiple ways to create folders and upload files into Azure Blob Container - you can use [Azure Portal](https://ms.portal.azure.com), [Storage Explorer](http://storageexplorer.com/), [Azure CLI2](/azure-cli-extension) or Azure SDK for your preferable programming language.In this example we will use Azure SDK for python to copy files into Blob.<jupyter_code>mnist_dataset_directory = 'mnist_dataset'
utils.dataset.download_and_upload_mnist_dataset_to_blob(
    blob_service, azure_blob_container_name, mnist_dataset_directory)<jupyter_output><empty_output><jupyter_text>Create Azure File ShareFor this example we will create a new File Share with name `batchaisample` under your storage account. This will be used to share the *training script file* and *output file*.**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you.<jupyter_code>azure_file_share_name = 'batchaisample'
file_service = FileService(cfg.storage_account_name, cfg.storage_account_key)
file_service.create_share(azure_file_share_name, fail_on_exist=False)<jupyter_output><empty_output><jupyter_text>Deploy Sample Script to Azure File ShareDownload original sample script<jupyter_code>script_to_deploy = 'resnet50_trainer.py'
utils.dataset.download_file('https://raw.githubusercontent.com/caffe2/caffe2/v0.6.0/caffe2/python/examples/resnet50_trainer.py', script_to_deploy)<jupyter_output><empty_output><jupyter_text>We will create a folder on Azure File Share containing a copy of original sample script<jupyter_code>script_directory = 'Caffe2Samples'
file_service.create_directory(
    azure_file_share_name, script_directory, fail_on_exist=False)
file_service.create_file_from_path(
    azure_file_share_name, script_directory, script_to_deploy, script_to_deploy)<jupyter_output><empty_output><jupyter_text>2. Create Azure Batch AI Compute Cluster  Configure Compute Cluster- For this example we will use a GPU cluster of `STANDARD_NC6` nodes. Number of nodes in the cluster is configured with `nodes_count` variable;- We will call the cluster `nc6`;So, the cluster will have the following parameters:<jupyter_code>nodes_count = 2
cluster_name = 'nc6'

parameters = models.ClusterCreateParameters(
    location=cfg.location,
    vm_size='STANDARD_NC6',
    scale_settings=models.ScaleSettings(
        manual=models.ManualScaleSettings(target_node_count=nodes_count)
    ),
    user_account_settings=models.UserAccountSettings(
        admin_user_name=cfg.admin,
        admin_user_password=cfg.<PASSWORD> or None,
        admin_user_ssh_public_key=cfg.admin_ssh_key or None,
    )
)<jupyter_output><empty_output><jupyter_text>Create Compute Cluster<jupyter_code>_ = client.clusters.create(cfg.resource_group, cfg.workspace, cluster_name, parameters).result()<jupyter_output><empty_output><jupyter_text>Monitor Cluster CreationGet the just created cluster. The `utilities` module contains a helper function to print out all kind of nodes count in the cluster.<jupyter_code>cluster = client.clusters.get(cfg.resource_group, cfg.workspace, cluster_name)
utils.cluster.print_cluster_status(cluster)<jupyter_output><empty_output><jupyter_text>3. Run Azure Batch AI Training Job  Configure Job- The job will use `caffe2ai/caffe2` container.- Will run `resnet50_trainer.py` from SCRIPT input directory;- Will output standard output and error streams to file share;- Will mount file share at folder with name `afs`. Full path of this folder on a computer node will be `$AZ_BATCHAI_JOB_MOUNT_ROOT/afs`;- Will mount Azure Blob Container at folder with name `bfs`. Full path of this folder on a computer node will be `$AZ_BATCHAI_JOB_MOUNT_ROOT/bfs`;- The job needs to know where to find mnist_replica.py and input MNIST dataset. We will create two input directories for this. The job will be able to reference those directories using environment variables:    - ```AZ_BATCHAI_INPUT_SCRIPT``` : refers to the directory containing the scripts at mounted Azure File Share     - ```AZ_BATCHAI_INPUT_DATASET``` : refers to the directory containing the training data on mounted Azure Blob Container- Will use $AZ_BATCHAI_SHARED_JOB_TEMP shared directory created by Batch AI to coordinate execution between nodes;- For demostration purpose, we will only run 5 epochs with epoch size as 2000.<jupyter_code>azure_file_share = 'afs'
azure_blob = 'bfs'
parameters = models.JobCreateParameters(
     location=cfg.location,
     cluster=models.ResourceId(id=cluster.id),
     node_count=2,
     mount_volumes=models.MountVolumes(
            azure_file_shares=[
                models.AzureFileShareReference(
                    account_name=cfg.storage_account_name,
                    credentials=models.AzureStorageCredentialsInfo(
                        account_key=cfg.storage_account_key),
                    azure_file_url='https://{0}.file.core.windows.net/{1}'.format(
                        cfg.storage_account_name, azure_file_share_name),
                    relative_mount_path=azure_file_share)
            ],
            azure_blob_file_systems=[
                models.AzureBlobFileSystemReference(
                    account_name=cfg.storage_account_name,
                    credentials=models.AzureStorageCredentialsInfo(
                        account_key=cfg.storage_account_key),
                    container_name=azure_blob_container_name,
                    relative_mount_path=azure_blob)
            ]
        ),
     input_directories = [
        models.InputDirectory(
            id='SCRIPT',
            path='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, script_directory)),
        models.InputDirectory(
            id='DATASET',
            path='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}/{1}'.format(azure_blob, mnist_dataset_directory))
    ],
    std_out_err_path_prefix='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}'.format(azure_file_share),
     container_settings=models.ContainerSettings(
         image_source_registry=models.ImageSourceRegistry(image='caffe2ai/caffe2')),
     caffe2_settings = models.Caffe2Settings(
         python_script_file_path='$AZ_BATCHAI_INPUT_SCRIPT/'+script_to_deploy,
         command_line_args='--num_shards 2 --shard_id $AZ_BATCHAI_TASK_INDEX --run_id 0 --epoch_size 2000 --num_epochs 5 --train_data $AZ_BATCHAI_INPUT_DATASET/mnist_train_lmdb --file_store_path $AZ_BATCHAI_SHARED_JOB_TEMP'))<jupyter_output><empty_output><jupyter_text>Create a training Job and wait for Job completion<jupyter_code>experiment_name = 'caffe2_experiment'
experiment = client.experiments.create(cfg.resource_group, cfg.workspace, experiment_name).result()
job_name = datetime.utcnow().strftime('caffe2_%m_%d_%Y_%H%M%S')
job = client.jobs.create(cfg.resource_group, cfg.workspace, experiment_name, job_name, parameters).result()
print('Created Job {0} in Experiment {1}'.format(job.name, experiment.name))<jupyter_output><empty_output><jupyter_text>Wait for Job to FinishThe job will start running when the cluster will have enough idle nodes. The following code waits for job to start running printing the cluster state. During job run, the code prints current content of stdout.txt.**Note** Execution may take several minutes to complete.<jupyter_code>utils.job.wait_for_job_completion(client, cfg.resource_group, cfg.workspace, 
                                  experiment_name, job_name, cluster_name, 'stdouterr', 'stderr-1.txt')<jupyter_output><empty_output><jupyter_text>List stdout.txt and stderr.txt files for the Job<jupyter_code>files = client.jobs.list_output_files(cfg.resource_group, cfg.workspace, experiment_name, job_name,
                                      models.JobsListOutputFilesOptions(outputdirectoryid='stdouterr')) 
for f in list(files):
    print(f.name, f.download_url or 'directory')<jupyter_output><empty_output><jupyter_text>4. Clean Up (Optional)  Delete the Job<jupyter_code>_ = client.jobs.delete(cfg.resource_group, cfg.workspace, experiment_name, job_name)<jupyter_output><empty_output><jupyter_text>Delete the ClusterWhen you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code.<jupyter_code>_ = client.clusters.delete(cfg.resource_group, cfg.workspace, cluster_name)<jupyter_output><empty_output><jupyter_text>Delete File ShareWhen you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code.<jupyter_code>service = FileService(cfg.storage_account_name, cfg.storage_account_key)
service.delete_share(azure_file_share_name)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup