<jupyter_start><jupyter_text>Quality Evaluation on ImageNet1k This notebooks allows to run the quality evaluation of any classsification model from [**CoreML Model Zoo**](https://github.com/vladimir-chernykh/coreml-model-zoo) on **ImageNet1k** data ([ILSVRC](http://www.image-net.org/challenges/LSVRC/)).The **validation set** is taken because it is almost never included into the training set of the specific model and has never changed since 2012.  Imports<jupyter_code>import os
import glob
import importlib

import xml.etree.ElementTree as ET

from tqdm.notebook import tqdm

import numpy as np
from joblib import Parallel, delayed

from PIL import Image
import matplotlib.pyplot as plt

from torch.utils.data import Dataset, DataLoader

import coremltools<jupyter_output>WARNING:root:TensorFlow version 2.2.0 detected. Last version known to be fully compatible is 1.14.0 .
WARNING:root:Keras version 2.3.1 detected. Last version known to be fully compatible of Keras is 2.2.4 .<jupyter_text>Define a function to dynamically import object from module<jupyter_code>def dynamic_import(abs_module_path, class_name):
    module_object = importlib.import_module(abs_module_path)

    target_class = getattr(module_object, class_name)

    return target_class<jupyter_output><empty_output><jupyter_text>Model Specify model name and version<jupyter_code>model_name = "mobilenet_v2"
model_spec = "mobilenet_v2_0.35_96_keras_applications"<jupyter_output><empty_output><jupyter_text>Load CoreML model<jupyter_code>model = coremltools.models.MLModel(f"../../vision/classification/{model_name}/models/{model_spec}.mlmodel")<jupyter_output><empty_output><jupyter_text>Data Read ImageNet classes info<jupyter_code>with open("./imagenet1k_classes.txt") as f:
    classes_coreml = f.readlines()
classes_coreml = np.array([c.rstrip('\n') for c in classes_coreml])
classes_coreml_dict = {c:i for i, c in enumerate(classes_coreml)}<jupyter_output><empty_output><jupyter_text>ImageNet dataset reader<jupyter_code>class DatasetImagenet(Dataset):
    
    def __init__(self, root, split, classes_mapping, transform=None):

        self.transform = transform
        self.root = root

        with open(classes_mapping) as fp:
            _temp = [line.rstrip('\n').split(" ", maxsplit=1) for line in fp]
            self.classnames_mapping = dict(_temp)
            self.classids_mapping = {wnid[0]: idd for idd, wnid in enumerate(_temp)}

        annotation_files = sorted(glob.glob(os.path.join(root, f"Annotations/CLS-LOC/{split}/*.xml")))

        self.image_files = np.array([file.replace("/Annotations/", "/Data/").rstrip(".xml") + ".JPEG" 
                                     for file in annotation_files])

        self.wnids = Parallel(n_jobs=8)(delayed(self._annotation_wnids_reader)(file) 
                                         for file in annotation_files)
        self.wnids = np.array([item[0] for item in self.wnids])
        self.classnames = np.array([self.classnames_mapping[wnid] for wnid in self.wnids])
        self.classids = np.array([self.classids_mapping[wnid] for wnid in self.wnids])

    @staticmethod
    def _annotation_wnids_reader(annotation_file):
        xmltree = ET.parse(annotation_file)
        objects = xmltree.findall("object")
        result = []
        for object_iter in objects:
            bndbox = object_iter.find("name")
            result.append(bndbox.text)
        return result

    def __len__(self):
        return len(self.classids)

    def __getitem__(self, idx):

        image = Image.open(self.image_files[idx]).convert("RGB")
        if self.transform is not None:
            image = self.transform(image)

        label = self.classids[idx]

        return image, label<jupyter_output><empty_output><jupyter_text>All the preprocessing steps are embedded into the network. All one has to do outside is resizing.Different models are trained in different frameworks and settings each of which requires a little bit different resizing options to fully repeat the quality.Load resizing transformation suitable exactly for the model of interest:<jupyter_code>%cd ../..
transform_resize_dict = dynamic_import(
    f"vision.classification.{model_name}.evaluation_utils.resize_utils", 
    "transform_resize_dict")
%cd tools/evaluation
transform_coreml = transform_resize_dict[model_spec]<jupyter_output>/Users/vovacher/Dropbox/coreml-zoo/coreml-model-zoo
/Users/vovacher/Dropbox/coreml-zoo/coreml-model-zoo/tools/evaluation<jupyter_text>Create `Dataset` object with ImageNet dataset. It might take around 10 seconds because all the `xml`s with annotations are parsed.<jupyter_code>dataset = DatasetImagenet(root="/Users/vovacher/Downloads/ILSVRC", 
                          split="val", 
                          classes_mapping="/Users/vovacher/Downloads/ILSVRC/LOC_synset_mapping.txt", 
                          transform=transform_coreml)<jupyter_output><empty_output><jupyter_text>Data sample example<jupyter_code>dataset[0]<jupyter_output><empty_output><jupyter_text>More examples visualized<jupyter_code>plt.figure(figsize=(20, 8))
for i in range(10):
    plt.subplot(2, 5, i + 1)
    plt.imshow(dataset[i][0])
    plt.axis("off")
    plt.title(classes_coreml[dataset[i][1]])<jupyter_output><empty_output><jupyter_text>Evaluation Define `DataLoader` which is able to sample from `Dataset` efficiently.<jupyter_code>def custom_collate_fn(batch):
    return {"input_1": batch[0][0], "target": batch[0][1]}
dataloader = DataLoader(dataset=dataset, 
                        batch_size=1, 
                        num_workers=4, 
                        collate_fn=custom_collate_fn)<jupyter_output><empty_output><jupyter_text>Evaluate the model on **ILSVRC 2012 validation** data<jupyter_code>y_true = []
y_preds_proba = []
for data in tqdm(dataloader):
    y_true.append(data["target"])
    _pred = model.predict(data=data, useCPUOnly=True)
    y_preds_proba.append(_pred)
y_true = np.array(y_true)
y_preds_proba = np.array(y_preds_proba)<jupyter_output><empty_output><jupyter_text>Transform predictions from class names into class numbers to compare with ground truth<jupyter_code>y_preds_proba_ids = []
for i in tqdm(range(len(y_preds_proba))):
    _outs = y_preds_proba[i]["output_1"]
    y_preds_proba_ids.append(np.zeros(len(_outs)))
    for k, v in _outs.items():
        y_preds_proba_ids[-1][classes_coreml_dict[k]] = v
y_preds_proba_ids = np.array(y_preds_proba_ids)<jupyter_output><empty_output><jupyter_text>Results Top-1 Accuracy<jupyter_code>top1_acc = (y_true == np.argmax(y_preds_proba_ids, axis=1)).mean()
top1_acc<jupyter_output><empty_output><jupyter_text>Top-5 Accuracy<jupyter_code>_top5 = np.argsort(y_preds_proba_ids, axis=1)[..., -5:]
top5_acc = np.mean([y_true[i] in _top5[i] for i in range(len(_top5))])
top5_acc<jupyter_output><empty_output>