<jupyter_start><jupyter_text>E2E ML on GCP: MLOps stage 4 : evaluation: get started with Vertex AI Model Evaluation             Run in Colab                        View on GitHub                        Open in Vertex AI Workbench                                                                                                       OverviewThis tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 4 : evaluation: get started with Vertex AI Model Evaluation.  Datasets**AutoML image model**The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip.**BigQuery ML tabular model**The dataset used for this tutorial is the Penguins dataset from [BigQuery public datasets](https://cloud.google.com/bigquery/public-data). This version of the dataset is used to predict the species of penguins from the available features like culmen-length, flipper-depth etc.**Custom model**This tutorial uses a pre-trained image classification model from TensorFlow Hub, which is trained on ImageNet dataset.Learn more about [ResNet V2 pretained model](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5). **Pipeline**BLAHThe dataset used for this tutorial is the [Bank Marketing](https://pantheon.corp.google.com/storage/browser/_details/cloud-ml-tables-data/bank-marketing.csv) . This dataset does not require any feature engineering. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket.  ObjectiveIn this tutorial, you learn how to use `Vertex AI Model Evaluation`.This tutorial uses the following Google Cloud ML services:- `Vertex AI AutoML`- `BigQuery ML`- `Vertex AI Training`- `Vertex AI Batch Prediction`- `Vertex AI Model Evaluation`- `Google Cloud Pipeline Components`The steps performed include:**SDK**- Evaluate an `AutoML` model.    - Train an `AutoML` image classification model.    - Retrieve the default evaluation metrics from training.    - Do a batch evaluation for a custom evaluation slice.- Evaluate a BigQuery ML model.    - Train a `BigQuery ML` tabular classification model.    - Retrieve the default evaluation metrics from training.    - Do a batch evaluation for a custom evaluation slice.- Evaluate a custom model.    - Do a batch evaluation for a custom evaluation slice.    - Add an evaluation to the `Model Registry` for the `Model` resource.    **Pipeline Components**- Evaluate an `AutoML` model.    - Train an `AutoML` image classification model.    - Retrieve the default evaluation metrics from training.    - Do a batch evaluation for a custom evaluation slice.- Evaluate a BigQuery ML model.    - Train a `BigQuery ML` tabular classification model.    - Retrieve the default evaluation metrics from training.    - Do a batch evaluation for a custom evaluation slice.- Evaluate a custom model.    - Do a batch evaluation for a custom evaluation slice.    - Add an evaluation to the `Model Registry` for the `Model` resource.  InstallationsInstall the packages required for executing this notebook.<jupyter_code>import os

# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)

# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"

# Install the packages
! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG -q
! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG -q
! pip3 install --upgrade google-cloud-bigquery $USER_FLAG -q
! pip3 install --upgrade tensorflow $USER_FLAG -q
! pip3 install --upgrade tensorflow-hub $USER_FLAG -q<jupyter_output><empty_output><jupyter_text>Restart the kernelOnce you've installed the additional packages, you need to restart the notebook kernel so it can find the packages.<jupyter_code>import os

if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)<jupyter_output><empty_output><jupyter_text>Before you begin GPU runtime*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU** Set up your Google Cloud project**The following steps are required, regardless of your notebook environment.**1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)3. [Enable the following APIs: Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,storage-component.googleapis.com)4. If you are running this notebook locally, you will need to install the [Cloud SDK]((https://cloud.google.com/sdk)).5. Enter your project ID in the cell below. Then run the  cell to make sure theCloud SDK uses the right project for all the commands in this notebook.**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`.  Set your project ID**If you don't know your project ID**, you may be able to get your project ID using `gcloud`.<jupyter_code>PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)
! gcloud config set project $PROJECT_ID<jupyter_output><empty_output><jupyter_text>RegionYou can also change the `REGION` variable, which is used for operationsthroughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.- Americas: `us-central1`- Europe: `europe-west4`- Asia Pacific: `asia-east1`You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations).<jupyter_code>REGION = "[your-region]"  # @param {type: "string"}

if REGION == "[your-region]":
    REGION = "us-central1"<jupyter_output><empty_output><jupyter_text>TimestampIf you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial.<jupyter_code>from datetime import datetime

TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")<jupyter_output><empty_output><jupyter_text>Authenticate your Google Cloud account**If you are using Vertex AI Workbench Notebooks**, your environment is already authenticated. Skip this step.**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.**Otherwise**, follow these steps:In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.**Click Create service account**.In the **Service account name** field, enter a name, and click **Create**.In the **Grant this service account access to project** section, click the Role drop-down list. Type "Vertex" into the filter box, and select **Vertex Administrator**. Type "Storage Object Admin" into the filter box, and select **Storage Object Admin**.Click Create. A JSON file that contains your key downloads to your local environment.Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.<jupyter_code># If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

import os
import sys

# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = False
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        IS_COLAB = True
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''<jupyter_output><empty_output><jupyter_text>Create a Cloud Storage bucket**The following steps are required, regardless of your notebook environment.**When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.<jupyter_code>BUCKET_NAME = "[your-bucket-name]"  # @param {type:"string"}
BUCKET_URI = f"gs://{BUCKET_NAME}"
if BUCKET_NAME == "" or BUCKET_NAME is None or BUCKET_NAME == "[your-bucket-name]":
    BUCKET_NAME = PROJECT_ID + "aip-" + TIMESTAMP
    BUCKET_URI = "gs://" + BUCKET_NAME<jupyter_output><empty_output><jupyter_text>**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket.<jupyter_code>! gsutil mb -l $REGION $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Finally, validate access to your Cloud Storage bucket by examining its contents:<jupyter_code>! gsutil ls -al $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Service Account**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below.<jupyter_code>SERVICE_ACCOUNT = "[your-service-account]"  # @param {type:"string"}
if (
    SERVICE_ACCOUNT == ""
    or SERVICE_ACCOUNT is None
    or SERVICE_ACCOUNT == "[your-service-account]"
):
    # Get your service account from gcloud
    if not IS_COLAB:
        shell_output = !gcloud auth list 2>/dev/null
        SERVICE_ACCOUNT = shell_output[2].replace("*", "").strip()

    if IS_COLAB:
        shell_output = ! gcloud projects describe  $PROJECT_ID
        project_number = shell_output[-1].split(":")[1].strip().replace("'", "")
        SERVICE_ACCOUNT = f"{<EMAIL>_<EMAIL>"

    print("Service Account:", SERVICE_ACCOUNT)<jupyter_output><empty_output><jupyter_text>Set service account access for Vertex AI PipelinesRun the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account.<jupyter_code>! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI

! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Set up variablesNext, set up some variables used throughout the tutorial. Import libraries and define constants<jupyter_code>import json

import google.cloud.aiplatform as aiplatform
import tensorflow as tf
import tensorflow_hub as hub
from kfp import dsl
from kfp.v2 import compiler
from kfp.v2.dsl import component<jupyter_output><empty_output><jupyter_text>Import BigQueryImport the BigQuery package into your Python environment.<jupyter_code>from google.cloud import bigquery<jupyter_output><empty_output><jupyter_text>Initialize Vertex AI SDK for PythonInitialize the Vertex AI SDK for Python for your project and corresponding bucket.<jupyter_code>aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)<jupyter_output><empty_output><jupyter_text>Create BigQuery clientCreate the BigQuery client.<jupyter_code>bqclient = bigquery.Client()<jupyter_output><empty_output><jupyter_text>Set hardware acceleratorsYou can set hardware accelerators for prediction.Set the variable `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:    (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)Otherwise specify `(None, None)` to use a container image to run on a CPU.Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locationsaccelerators) hardware accelerator support for your region<jupyter_code>import os

if os.getenv("IS_TESTING_DEPLOY_GPU"):
    DEPLOY_GPU, DEPLOY_NGPU = (
        aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,
        int(os.getenv("IS_TESTING_DEPLOY_GPU")),
    )
else:
    DEPLOY_GPU, DEPLOY_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)<jupyter_output><empty_output><jupyter_text>Set pre-built containersSet the pre-built Docker container image for prediction.- Set the variable `TF` to the TensorFlow version of the container image. For example, `2-1` would be version 2.1, and `1-15` would be version 1.15. The following list shows some of the pre-built images available:For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers).<jupyter_code>if os.getenv("IS_TESTING_TF"):
    TF = os.getenv("IS_TESTING_TF")
else:
    TF = "2-5".replace(".", "-")

if TF[0] == "2":
    if DEPLOY_GPU:
        DEPLOY_VERSION = "tf2-gpu.{}".format(TF)
    else:
        DEPLOY_VERSION = "tf2-cpu.{}".format(TF)
else:
    if DEPLOY_GPU:
        DEPLOY_VERSION = "tf-gpu.{}".format(TF)
    else:
        DEPLOY_VERSION = "tf-cpu.{}".format(TF)

DEPLOY_IMAGE = "{}-docker.pkg.dev/vertex-ai/prediction/{}:latest".format(
    REGION.split("-")[0], DEPLOY_VERSION
)

print("Deployment:", DEPLOY_IMAGE, DEPLOY_GPU)<jupyter_output><empty_output><jupyter_text>Set machine typeNext, set the machine type to use for prediction.- Set the variable `DEPLOY_COMPUTE` to configure the compute resources for the VM you will use for prediction. - `machine type`     - `n1-standard`: 3.75GB of memory per vCPU.     - `n1-highmem`: 6.5GB of memory per vCPU     - `n1-highcpu`: 0.9 GB of memory per vCPU - `vCPUs`: number of \[2, 4, 8, 16, 32, 64, 96 \]*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*<jupyter_code>if os.getenv("IS_TESTING_DEPLOY_MACHINE"):
    MACHINE_TYPE = os.getenv("IS_TESTING_DEPLOY_MACHINE")
else:
    MACHINE_TYPE = "n1-standard"

VCPU = "4"
DEPLOY_COMPUTE = MACHINE_TYPE + "-" + VCPU
print("Deploy machine type", DEPLOY_COMPUTE)<jupyter_output><empty_output><jupyter_text>Introduction to Vertex AI Model Evaluation for AutoML models.For AutoML models, you can retrieve the model evaluation metrics that were obtained during training from the dataset split into train and test, using the `Vertex AI Model Evaluation` service. Additionally, you can further evaluate the model with custom evaluation slices.  Location of Cloud Storage training data.Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage.<jupyter_code>IMPORT_FILE = (
    "gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv"
)<jupyter_output><empty_output><jupyter_text>Create the DatasetNext, create the `Dataset` resource using the `create` method for the `ImageDataset` class, which takes the following parameters:- `display_name`: The human readable name for the `Dataset` resource.- `gcs_source`: A list of one or more dataset index files to import the data items into the `Dataset` resource.- `import_schema_uri`: The data labeling schema for the data items.This operation may take several minutes.<jupyter_code>dataset = aiplatform.ImageDataset.create(
    display_name="Flowers" + "_" + TIMESTAMP,
    gcs_source=[IMPORT_FILE],
    import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification,
)

print(dataset.resource_name)<jupyter_output><empty_output><jupyter_text>Create and run training pipelineTo train an AutoML model, you perform two steps: 1) create a training pipeline, and 2) run the pipeline. Create training pipelineAn AutoML training pipeline is created with the `AutoMLImageTrainingJob` class, with the following parameters:- `display_name`: The human readable name for the `TrainingJob` resource.- `prediction_type`: The type task to train the model for.  - `classification`: An image classification model.  - `object_detection`: An image object detection model.- `multi_label`: If a classification task, whether single (`False`) or multi-labeled (`True`).- `model_type`: The type of model for deployment.  - `CLOUD`: Deployment on Google Cloud  - `CLOUD_HIGH_ACCURACY_1`: Optimized for accuracy over latency for deployment on Google Cloud.  - `CLOUD_LOW_LATENCY_`: Optimized for latency over accuracy for deployment on Google Cloud.  - `MOBILE_TF_VERSATILE_1`: Deployment on an edge device.  - `MOBILE_TF_HIGH_ACCURACY_1`:Optimized for accuracy over latency for deployment on an edge device.  - `MOBILE_TF_LOW_LATENCY_1`: Optimized for latency over accuracy for deployment on an edge device.- `base_model`: (optional) Transfer learning from existing `Model` resource -- supported for image classification only.The instantiated object is the DAG (directed acyclic graph) for the training job.<jupyter_code>dag = aiplatform.AutoMLImageTrainingJob(
    display_name="flowers_" + TIMESTAMP,
    prediction_type="classification",
    multi_label=False,
    model_type="CLOUD",
    base_model=None,
)

print(dag)<jupyter_output><empty_output><jupyter_text>Run the training pipelineNext, you run the DAG to start the training job by invoking the method `run`, with the following parameters:- `dataset`: The `Dataset` resource to train the model.- `model_display_name`: The human readable name for the trained model.- `training_fraction_split`: The percentage of the dataset to use for training.- `test_fraction_split`: The percentage of the dataset to use for test (holdout data).- `validation_fraction_split`: The percentage of the dataset to use for validation.- `budget_milli_node_hours`: (optional) Maximum training time specified in unit of millihours (1000 = hour).- `disable_early_stopping`: If `True`, training maybe completed before using the entire budget if the service believes it cannot further improve on the model objective measurements.The `run` method when completed returns the `Model` resource.The execution of the training pipeline will take upto 20 minutes.<jupyter_code>model = dag.run(
    dataset=dataset,
    model_display_name="flowers_" + TIMESTAMP,
    training_fraction_split=0.8,
    validation_fraction_split=0.1,
    test_fraction_split=0.1,
    budget_milli_node_hours=8000,
    disable_early_stopping=False,
)<jupyter_output><empty_output><jupyter_text>Retrieving the default evaluation for `AutoML Model` resourceBLAH GAPIC PlaceholderAfter your model has finished training, you can review the evaluation scores for it.First, you need to get a reference to the new model. As with datasets, you can either use the reference to the model variable you created when you deployed the model or you can list all of the models in your project.<jupyter_code>model_evaluations = model.list_model_evaluations()

for model_evaluation in model_evaluations:
    print(model_evaluation.to_dict())<jupyter_output><empty_output><jupyter_text>Evaluating on a custom evaluation slicePLACEHOLDER - BLAH  Make the batch input fileNow make a batch input file, which you store in your local Cloud Storage bucket. The batch input file must be in JSONL format. In For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:- `content`: The Cloud Storage path to the image.- `mime_type`: The content type. In our example, it is a `jpeg` file.For example:                        {'content': '[your-bucket]/file1.jpg', 'mime_type': 'jpeg'}    For demonstration purposes, to create an evaluation slice, you use a portion of the training data -- as if it was separate (non-training) data, such as instances seen in production.<jupyter_code>EVAL_SLICE = BUCKET_URI + "/flowers_eval.jsonl"

! gsutil cat {IMPORT_FILE} | head -n 200 >tmp.csv

import csv

entries = []
with open("tmp.csv", "r") as f:
    reader = csv.reader(f)
    for row in reader:
        path = row[0]
        label = row[1]

        file = path.split("/")[-1]

        new_path = BUCKET_URI + "/flowers/" + file

        ! gsutil cp {path} {new_path} >/dev/null
        entries.append({"content": new_path, "mime_type": "jpeg"})

import json

with open("tmp.jsonl", "w") as f:
    for entry in entries:
        f.write(json.dumps(entry) + "\n")

! gsutil cp tmp.jsonl {EVAL_SLICE}
#! rm tmp.csv tmp.jsonl<jupyter_output><empty_output><jupyter_text>Make the batch prediction requestNow that your Model resource is trained, you can make a batch prediction by invoking the batch_predict() method, with the following parameters:- `job_display_name`: The human readable name for the batch prediction job.- `instances_format`: The format of the prediction request; can only be JSONL (default).- `gcs_source`: A list of one or more batch request input files.- `gcs_destination_prefix`: The Cloud Storage location for storing the batch prediction resuls.- `sync`: If set to True, the call will block while waiting for the asynchronous batch job to complete.<jupyter_code>batch_predict_job = model.batch_predict(
    job_display_name="flowers_" + TIMESTAMP,
    instances_format="jsonl",
    gcs_source=EVAL_SLICE,
    gcs_destination_prefix=BUCKET_URI,
    sync=True,
)

print(batch_predict_job)<jupyter_output><empty_output><jupyter_text>TODO: Get batch results  Get the predictionsNext, get the results from the completed batch prediction job.The results are written to the Cloud Storage output bucket you specified in the batch prediction request. You call the method iter_outputs() to get a list of each Cloud Storage file generated with the results. Each file contains one or more prediction requests in a JSON format:- `content`: The prediction request.- `prediction`: The prediction response. - `ids`: The internal assigned unique identifiers for each prediction request. - `displayNames`: The class names for each class label. - `confidences`: The predicted confidence, between 0 and 1, per class label.<jupyter_code>import json

import tensorflow as tf

bp_iter_outputs = batch_predict_job.iter_outputs()

prediction_results = list()
for blob in bp_iter_outputs:
    if blob.name.split("/")[-1].startswith("prediction"):
        prediction_results.append(blob.name)

tags = list()
for prediction_result in prediction_results:
    gfile_name = f"gs://{bp_iter_outputs.bucket.name}/{prediction_result}"
    with tf.io.gfile.GFile(name=gfile_name, mode="r") as gfile:
        for line in gfile.readlines():
            line = json.loads(line)
            print(line)
            break<jupyter_output><empty_output><jupyter_text>Delete temporary resourcesNext, you delete all the temporary resources created by this example.<jupyter_code>try:
    dag.delete()
    model.delete()
    batch_predict_job.delete()
except Exception as e:
    print(e)<jupyter_output><empty_output><jupyter_text>Introduction to Vertex AI Model Evaluation for BigQuery ML models.For BigQuery ML models, you can retrieve the model evaluation metrics that were obtained during training from the dataset split into train and test, using the `Vertex AI Model Evaluation` service. Additionally, you can further evaluate the model with custom evaluation slices.  Location of the BigQuery training dataNow set the variable `IMPORT_FILE` and `BQ_TABLE` to the location of the training data in `BigQuery`.<jupyter_code>IMPORT_FILE = "bq://bigquery-public-data.ml_datasets.penguins"
BQ_TABLE = "bigquery-public-data.ml_datasets.penguins"<jupyter_output><empty_output><jupyter_text>Create BQ dataset resourceFirst, you create an empty dataset resource in your project.<jupyter_code>BQ_DATASET_NAME = "penguins"
DATASET_QUERY = f"""CREATE SCHEMA {BQ_DATASET_NAME}
"""

job = bqclient.query(DATASET_QUERY)<jupyter_output><empty_output><jupyter_text>Setting permissions to automatically register the modelYou need to set some additional IAM permissions for BigQuery ML to automatically upload and register the model after training. Depending on your service account, the setting of the permissions below may fail. In this case, we recommend executing the permissions in a Cloud Shell.<jupyter_code>! gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member='serviceAccount:<EMAIL>' \
      --role='roles/aiplatform.admin'

! gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member='user:<EMAIL>' \
      --role='roles/aiplatform.admin'<jupyter_output><empty_output><jupyter_text>Training and registering the BigQuery ML modelNext, you create and train a BigQuery ML tabular classification model from the public dataset penguins and store the model in your project using the `CREATE MODEL` statement. The model configuration is specified in the `OPTIONS` statement as follows:- `model_type`: The type and archictecture of tabular model to train, e.g., DNN classification.- `labels`: The column which are the labels.- `model_registry`: Set to "vertex_ai" to indicate automatic registation to `Vertex AI Model Registry`.- `vertex_ai_model_id`: The human readable display name for the registered model.- `vertex_ai_model_version_aliases`: Alternate names for the model.Learn more about [The CREATE MODEL statement](https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-create).<jupyter_code>MODEL_NAME = "penguins"
MODEL_QUERY = f"""
CREATE OR REPLACE MODEL `{BQ_DATASET_NAME}.{MODEL_NAME}`
OPTIONS(
    model_type='DNN_CLASSIFIER',
    labels = ['species'],
    model_registry="vertex_ai",
    vertex_ai_model_id="bqml_model_{TIMESTAMP}", 
    vertex_ai_model_version_aliases=["1"]
    )
AS
SELECT *
FROM `{BQ_TABLE}`
"""

job = bqclient.query(MODEL_QUERY)
print(job.errors, job.state)

while job.running():
    from time import sleep

    sleep(30)
    print("Running ...")
print(job.errors, job.state)

tblname = job.ddl_target_table
tblname = "{}.{}".format(tblname.dataset_id, tblname.table_id)
print("{} created in {}".format(tblname, job.ended - job.started))<jupyter_output><empty_output><jupyter_text>Evaluate the trained BigQuery model using BigQueryNext, retrieve the model evaluation from within BigQuery for the trained BigQuery ML model.Learn more about [The ML.EVALUATE function](https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-evaluate).<jupyter_code>EVAL_QUERY = f"""
SELECT *
FROM
  ML.EVALUATE(MODEL {BQ_DATASET_NAME}.{MODEL_NAME})
ORDER BY  roc_auc desc
LIMIT 1"""

job = bqclient.query(EVAL_QUERY)
results = job.result().to_dataframe()
print(results)<jupyter_output><empty_output><jupyter_text>Find the model in the `Vertex AI Model Registry`Finally, you can use the `Vertex AI Model` list() method with a filter query to find the automatically registered model.<jupyter_code>models = aiplatform.Model.list(filter="display_name=bqml_model_" + TIMESTAMP)
model = models[0]

print(model.gca_resource)<jupyter_output><empty_output><jupyter_text>Retrieving the default evaluation for `BigQuery ML Model` resource from `Vertex AI Model Registry`BLAH GAPIC PlaceholderAfter your model has finished training, you can review the evaluation scores for it.First, you need to get a reference to the new model. As with datasets, you can either use the reference to the model variable you created when you deployed the model or you can list all of the models in your project.<jupyter_code># Get a reference to the Model Service client
client_options = {"api_endpoint": f"{REGION}-aiplatform.googleapis.com"}
model_service_client = aiplatform.gapic.ModelServiceClient(
    client_options=client_options
)

model_evaluations = model_service_client.list_model_evaluations(
    parent=model.resource_name
)
model_evaluation = list(model_evaluations)[0]
print(model_evaluation)<jupyter_output><empty_output><jupyter_text>Evaluating on a custom evaluation slicePLACEHOLDER - BLAHBLAH - BATCH FILE FORMAT  Delete temporary resourcesNext, you delete all the temporary resources created by this example.<jupyter_code>try:
    model.delete()
    batch_predict_job.delete()
except Exception as e:
    print(e)

try:
    # Delete the created BigQuery dataset
    ! bq rm -r -f $PROJECT_ID:$BQ_DATASET_NAME
except Exception as e:
    print(e)

MODEL_QUERY = f"""
DROP MODEL `{BQ_DATASET_NAME}.{MODEL_NAME}`
"""

job = bqclient.query(MODEL_QUERY)<jupyter_output><empty_output><jupyter_text>Introduction to Vertex AI Model Evaluation for custom models.For custom models, you can retrieve the model evaluation metrics that were BLAHobtained during training from the dataset split into train and test, using the `Vertex AI Model Evaluation` service. Additionally, you can further evaluate the model with custom evaluation slices.  Get pretrained model from TensorFlow HubFor demonstration purposes, this tutorial uses a pretrained model from TensorFlow Hub (TFHub), which is then uploaded to a `Vertex AI Model` resource. Once you have a `Vertex AI Model` resource, the model can be deployed to a `Vertex AI Endpoint` resource. Download the pretrained modelFirst, you download the pretrained model from TensorFlow Hub. The model gets downloaded as a TF.Keras layer. To finalize the model, in this example, you create a `Sequential()` model with the downloaded TFHub model as a layer, and specify the input shape to the model.<jupyter_code>tfhub_model = tf.keras.Sequential(
    [hub.KerasLayer("https://tfhub.dev/google/imagenet/resnet_v2_101/classification/5")]
)

tfhub_model.build([None, 224, 224, 3])

tfhub_model.summary()<jupyter_output><empty_output><jupyter_text>Save the model artifactsAt this point, the model is in memory. Next, you save the model artifacts to a Cloud Storage location.*Note:* For TF Serving, the MODEL_DIR must end in a subfolder that is a number, e.g., 1.<jupyter_code>MODEL_DIR = BUCKET_URI + "/model/1"
tfhub_model.save(MODEL_DIR)<jupyter_output><empty_output><jupyter_text>Upload the model for servingNext, you will upload your TF.Keras model from the custom job to Vertex `Model` service, which will create a Vertex `Model` resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex AI, your serving function ensures that the data is decoded on the model server before it is passed as input to your model. How does the serving function workWhen you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.The serving function consists of two parts:- `preprocessing function`:  - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).  - Performs the same preprocessing of the data that was done during training the underlying model -- e.g., normalizing, scaling, etc.- `post-processing function`:  - Converts the model output to format expected by the receiving application -- e.q., compresses the output.  - Packages the output for the the receiving application -- e.g., add headings, make JSON object, etc.Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you cannot use TF graph operations that require a dynamic graph. If you do, you will get an error during the compile of the serving function which will indicate that you are using an EagerTensor which is not supported.  Serving function for image data PreprocessingTo pass images to the prediction service, you encode the compressed (e.g., JPEG) image bytes into base 64 -- which makes the content safe from modification while transmitting binary data over the network. Since this deployed model expects input data as raw (uncompressed) bytes, you need to ensure that the base 64 encoded data gets converted back to raw bytes, and then preprocessed to match the model input requirements, before it is passed as input to the deployed model.To resolve this, you define a serving function (`serving_fn`) and attach it to the model as a preprocessing step. Add a `@tf.function` decorator so the serving function is fused to the underlying model (instead of upstream on a CPU).When you send a prediction or explanation request, the content of the request is base 64 decoded into a Tensorflow string (`tf.string`), which is passed to the serving function (`serving_fn`). The serving function preprocesses the `tf.string` into raw (uncompressed) numpy bytes (`preprocess_fn`) to match the input requirements of the model:- `io.decode_jpeg`- Decompresses the JPG image which is returned as a Tensorflow tensor with three channels (RGB).- `image.convert_image_dtype` - Changes integer pixel values to float 32, and rescales pixel data between 0 and 1.- `image.resize` - Resizes the image to match the input shape for the model.At this point, the data can be passed to the model (`m_call`), via a concrete function. The serving function is a static graph, while the model is a dynamic graph. The concrete function performs the tasks of marshalling the input data from the serving function to the model, and marshalling the prediction result from the model back to the serving function.<jupyter_code>CONCRETE_INPUT = "numpy_inputs"


def _preprocess(bytes_input):
    decoded = tf.io.decode_jpeg(bytes_input, channels=3)
    decoded = tf.image.convert_image_dtype(decoded, tf.float32)
    resized = tf.image.resize(decoded, size=(224, 224))
    return resized


@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])
def preprocess_fn(bytes_inputs):
    decoded_images = tf.map_fn(
        _preprocess, bytes_inputs, dtype=tf.float32, back_prop=False
    )
    return {
        CONCRETE_INPUT: decoded_images
    }  # User needs to make sure the key matches model's input


@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])
def serving_fn(bytes_inputs):
    images = preprocess_fn(bytes_inputs)
    prob = m_call(**images)
    return prob


m_call = tf.function(tfhub_model.call).get_concrete_function(
    [tf.TensorSpec(shape=[None, 224, 224, 3], dtype=tf.float32, name=CONCRETE_INPUT)]
)

tf.saved_model.save(tfhub_model, MODEL_DIR, signatures={"serving_default": serving_fn})<jupyter_output><empty_output><jupyter_text>Get the serving function signatureYou can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.For your purpose, you need the signature of the serving function. Why? Well, when we send our data for prediction as a HTTP request packet, the image data is base64 encoded, and our TF.Keras model takes numpy input. Your serving function will do the conversion from base64 to a numpy array.When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request.<jupyter_code>loaded = tf.saved_model.load(MODEL_DIR)

serving_input = list(
    loaded.signatures["serving_default"].structured_input_signature[1].keys()
)[0]
print("Serving function input:", serving_input)<jupyter_output><empty_output><jupyter_text>Upload the TensorFlow Hub model to a `Vertex AI Model` resourceFinally, you upload the model artifacts from the TFHub model into a `Vertex AI Model` resource.<jupyter_code>model = aiplatform.Model.upload(
    display_name="example_" + TIMESTAMP,
    artifact_uri=MODEL_DIR,
    serving_container_image_uri=DEPLOY_IMAGE,
)

print(model)<jupyter_output><empty_output><jupyter_text>BLAH Do batch prediction on custom model  BLAH register the custom evaluation metrics  Model evaluation using `Vertex AI Pipeline` componentsIn this section, you perform model evaluations on `AutoML`, `BigQuery ML` and custom models using `Vertex AI Pipeline` components  AutoML model evaluation pipeline componentBLAHAdditionally, you can evaluate an AutoML model with custom evaluation slices using the combination of BatchPredictionOp and ModelEvaluationOp components, as:- The custom evaluation slice data contains the label values (ground truths).- Perform a batch prediction on the custom evaluation slice.- Perform a model evaluation with the batch prediction results and label values.  Location of Cloud Storage training data.Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage.<jupyter_code>IMPORT_FILE = "gs://cloud-ml-tables-data/bank-marketing.csv"
! gsutil cat {IMPORT_FILE} | head -n 40000 > train.csv
! gsutil cat {IMPORT_FILE} | head -n 1 >eval.csv
! gsutil cat {IMPORT_FILE} | tail -n 5200 >> eval.csv

IMPORT_TRAIN = BUCKET_NAME + "/train.csv"
IMPORT_EVAL = BUCKET_NAME + "/eval.csv"

! gsutil cp train.csv {IMPORT_TRAIN}
! gsutil cp eval.csv {IMPORT_EVAL}

! rm -f train.csv eval.csv<jupyter_output><empty_output><jupyter_text>Create AutoML model evaluation componentThe Vertex AI pre-built pipeline components does not currently have a component for retrieiving the model evaluations for a AutoML model. So, you will first write your own component, as follows:- Takes as input the region and Model artifacts returned from an AutoML training component.- Create a client interface to the Vertex AI Model service (`metadata["resource_name"]).- Construct the resource ID for the model from the model artifact parameter.- Retrieve the model evaluation- Return the model evaluation as a string.<jupyter_code>from kfp.v2.dsl import Artifact, Input, Model


@component(packages_to_install=["google-cloud-aiplatform"])
def evaluateAutoMLModelOp(model: Input[Artifact], region: str) -> str:
    import logging

    import google.cloud.aiplatform.gapic as gapic

    # Get a reference to the Model Service client
    client_options = {"api_endpoint": f"{region}-aiplatform.googleapis.com"}
    model_service_client = gapic.ModelServiceClient(client_options=client_options)

    model_id = model.metadata["resourceName"]

    model_evaluations = model_service_client.list_model_evaluations(parent=model_id)
    model_evaluation = list(model_evaluations)[0]
    logging.info(model_evaluation)
    return str(model_evaluation)<jupyter_output><empty_output><jupyter_text>Construct pipeline for AutoML training, and batch model evaluationNext, construct the pipeline with the following tasks:- Create a Vertex AI Dataset resource.- Train a AutoML tabular classification model.- Retrieve the AutoML evaluation statistics.- Make a batch prediction with the AutoML model, using an evaluation slice that was not used during training.- Evaluate the AutoML model using the results from the batch prediction.<jupyter_code>PIPELINE_ROOT = "{}/pipeline_root/automl_lbn_training".format(BUCKET_NAME)


@dsl.pipeline(
    name="automl-lbn-training", description="AutoML tabular classification training"
)
def pipeline(
    import_file: str,
    batch_files: list,
    display_name: str,
    bucket: str = PIPELINE_ROOT,
    project: str = PROJECT_ID,
    region: str = REGION,
):
    from google_cloud_pipeline_components import aiplatform as gcc_aip
    from google_cloud_pipeline_components.experimental.evaluation import \
        ModelEvaluationOp
    from google_cloud_pipeline_components.v1.batch_predict_job import \
        ModelBatchPredictOp

    dataset_op = gcc_aip.TabularDatasetCreateOp(
        project=project, display_name=display_name, gcs_source=import_file
    )

    training_op = gcc_aip.AutoMLTabularTrainingJobRunOp(
        project=project,
        display_name=display_name,
        optimization_prediction_type="classification",
        dataset=dataset_op.outputs["dataset"],
        model_display_name=display_name,
        training_fraction_split=0.8,
        validation_fraction_split=0.1,
        test_fraction_split=0.1,
        budget_milli_node_hours=8000,
        optimization_objective="minimize-log-loss",
        target_column="Deposit",
    )

    eval_op = evaluateAutoMLModelOp(model=training_op.outputs["model"], region=region)

    batch_op = ModelBatchPredictOp(
        project=project,
        job_display_name="batch_predict_job",
        model=training_op.outputs["model"],
        gcs_source_uris=batch_files,
        gcs_destination_output_uri_prefix=bucket,
        instances_format="csv",
        predictions_format="jsonl",
        model_parameters={},
        machine_type=DEPLOY_COMPUTE,
        starting_replica_count=1,
        max_replica_count=1,
    ).after(eval_op)

    batch_eval_op = ModelEvaluationOp(
        project=project,
        root_dir=bucket,
        problem_type="classification",
        classification_type="multiclass",
        ground_truth_column="Deposit",
        class_names=["0", "1"],
        predictions_format="jsonl",
        batch_prediction_job=batch_op.outputs["batchpredictionjob"],
    )<jupyter_output><empty_output><jupyter_text>Compile and execute the AutoML training, and batch model evaluation pipelineNext, you compile the pipeline and then execute it. The pipeline takes the following parameters, which are passed as the dictionary `parameter_values`:- `import_file`: The Cloud Storage location of the training data.- `batch_files`: A list of one or more Cloud Storage locations of evaluation data.- `display_name`: Display name for Vertex AI Model and Endpoint resources.- `project`: The project ID.- `region`: The region.<jupyter_code>compiler.Compiler().compile(
    pipeline_func=pipeline, package_path="automl_lbn_training.json"
)

pipeline = aip.PipelineJob(
    display_name="automl_lbn_training",
    template_path="automl_lbn_training.json",
    pipeline_root=PIPELINE_ROOT,
    parameter_values={
        "import_file": IMPORT_TRAIN,
        "batch_files": [IMPORT_EVAL],
        "display_name": "bank" + TIMESTAMP,
        "project": PROJECT_ID,
        "region": REGION,
    },
)

pipeline.run()

! rm -f automl_lbn_training.json<jupyter_output><empty_output><jupyter_text>View the AutoML training and batch evaluation pipeline results<jupyter_code>PROJECT_NUMBER = pipeline.gca_resource.name.split("/")[1]
print(PROJECT_NUMBER)


def print_pipeline_output(job, output_task_name):
    JOB_ID = job.name
    print(JOB_ID)
    for _ in range(len(job.gca_resource.job_detail.task_details)):
        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id
        EXECUTE_OUTPUT = (
            PIPELINE_ROOT
            + "/"
            + PROJECT_NUMBER
            + "/"
            + JOB_ID
            + "/"
            + output_task_name
            + "_"
            + str(TASK_ID)
            + "/executor_output.json"
        )
        GCP_RESOURCES = (
            PIPELINE_ROOT
            + "/"
            + PROJECT_NUMBER
            + "/"
            + JOB_ID
            + "/"
            + output_task_name
            + "_"
            + str(TASK_ID)
            + "/gcp_resources"
        )
        if tf.io.gfile.exists(EXECUTE_OUTPUT):
            ! gsutil cat $EXECUTE_OUTPUT
            break
        elif tf.io.gfile.exists(GCP_RESOURCES):
            ! gsutil cat $GCP_RESOURCES
            break

    return EXECUTE_OUTPUT


print("tabular-dataset-create")
artifacts = print_pipeline_output(pipeline, "tabular-dataset-create")
print("\n\n")
print("automl-tabular-training-job")
artifacts = print_pipeline_output(pipeline, "automl-tabular-training-job")
print("\n\n")
print("evaluateautomlmodelop")
artifacts = print_pipeline_output(pipeline, "evaluateautomlmodelop")
output = !gsutil cat $artifacts
output = json.loads(output[0])
metrics = output["parameters"]["Output"]["stringValue"]
print("\n")
print(metrics)
print("\n\n")
print("model-batch-predict")
artifacts = print_pipeline_output(pipeline, "model-batch-predict")
output = !gsutil cat $artifacts
output = json.loads(output[0])
print("\n\n")
print(
    output["artifacts"]["batchpredictionjob"]["artifacts"][0]["metadata"][
        "gcsOutputDirectory"
    ]
)
print("model-evaluation")
artifacts = print_pipeline_output(pipeline, "model-evaluation")<jupyter_output><empty_output><jupyter_text>Delete a pipeline jobAfter a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`.<jupyter_code>pipeline.delete()<jupyter_output><empty_output><jupyter_text>Introduction to Vertex AI Model Evaluation for BigQuery ML models.For BigQuery ML models, you can retrieve the model evaluation metrics that were obtained during training from the dataset split into train and test, using the `BigQuery ML` service.Additionally, you can evaluate an BigQuery ML model with custom evaluation slices using the combination of BLAH`BatchPredictionOp` and `ModelEvaluationOp` components, as:    - The custom evaluation slice data contains the label values (ground truths).    - Perform a batch prediction on the custom evaluation slice.    - Perform a model evaluation with the batch prediction results and label values.<jupyter_code>IMPORT_FILE = "bq://bigquery-public-data.ml_datasets.penguins"
BQ_TABLE = "bigquery-public-data.ml_datasets.penguins"
BQ_TABLE = "bigquery-public-data.ml_datasets.penguins"
BQ_DATASET = BQ_TABLE.split(".")[1]


def get_data(slice_name, limit):
    query = f"""
    CREATE OR REPLACE TABLE `{slice_name}`
    AS (
        WITH
          penguins AS (
          SELECT
            island,
            sex,
            culmen_length_mm,
            culmen_depth_mm,
            flipper_length_mm,
            body_mass_g,
            species
          FROM
            `{BQ_TABLE}`
        )

        SELECT
          island,
          sex,
          culmen_length_mm,
          culmen_depth_mm,
          flipper_length_mm,
          body_mass_g,
          species
        FROM
          penguins
        LIMIT {limit}
    )
    """

    response = bqclient.query(query)
    _ = response.result()


BQ_TABLE_EVAL = f"{PROJECT_ID}.{BQ_DATASET}.penguins_eval"
IMPORT_EVAL = f"bq://{BQ_TABLE_EVAL}"
LIMIT = 44
get_data(BQ_TABLE_EVAL, LIMIT)

BQ_TABLE_TRAIN = f"{PROJECT_ID}.{BQ_DATASET}.penguins_train"
IMPORT_TRAIN = f"bq://{BQ_TABLE_TRAIN}"
LIMIT = "300 OFFSET 44"
get_data(BQ_TABLE_TRAIN, LIMIT)<jupyter_output><empty_output><jupyter_text>Construct pipeline for BigQuery ML training, and batch model evaluationNext, construct the pipeline with the following tasks:- Create a BigQuery ML Dataset resource.- Train a BigQuery ML tabular classification model.- Retrieve the BigQuery ML evaluation statistics.- Make a batch prediction with the BigQuery ML model, using an evaluation slice that was not used during training.- Evaluate the BigQuery ML model using the results from the batch prediction.<jupyter_code>PIPELINE_ROOT = f"{BUCKET_NAME}/bq_query"


@dsl.pipeline(name="bq-hello-world", pipeline_root=PIPELINE_ROOT)
def pipeline(
    bq_train_table: str,
    bq_eval_table: str,
    label: str,
    class_names: list,
    dataset: str,
    model: str,
    artifact_uri: str,
    # num_trials: int,
    deploy_image: str,
    machine_type: str,
    min_replica_count: int,
    max_replica_count: int,
    display_name: str,
    bucket: str,
    accelerator_type: str = "",
    accelerator_count: int = 0,
    project: str = PROJECT_ID,
    location: str = "US",
    region: str = "us-central1",
):
    from google_cloud_pipeline_components.experimental.evaluation import \
        ModelEvaluationOp
    from google_cloud_pipeline_components.v1.batch_predict_job import \
        ModelBatchPredictOp
    from google_cloud_pipeline_components.v1.bigquery import (
        BigqueryCreateModelJobOp, BigqueryEvaluateModelJobOp,
        BigqueryExportModelJobOp, BigqueryQueryJobOp)
    from google_cloud_pipeline_components.v1.model import ModelUploadOp

    bq_dataset = BigqueryQueryJobOp(
        project=project, location="US", query=f"CREATE SCHEMA {dataset}"
    )

    bq_model = BigqueryCreateModelJobOp(
        project=project,
        location=location,
        query=f"CREATE OR REPLACE MODEL {dataset}.{model} OPTIONS (model_type='dnn_classifier', labels=['{label}']) AS SELECT * FROM `{bq_train_table}` WHERE body_mass_g IS NOT NULL AND sex IS NOT NULL",
    ).after(bq_dataset)

    bq_eval = BigqueryEvaluateModelJobOp(
        project=PROJECT_ID, location="US", model=bq_model.outputs["model"]
    ).after(bq_model)

    bq_export = BigqueryExportModelJobOp(
        project=project,
        location=location,
        model=bq_model.outputs["model"],
        model_destination_path=artifact_uri,
    ).after(bq_model)

    model_upload = ModelUploadOp(
        display_name=display_name,
        artifact_uri=artifact_uri,
        serving_container_image_uri=deploy_image,
        project=project,
        location=region,
    ).after(bq_export)

    batch_predict = ModelBatchPredictOp(
        project=project,
        job_display_name="batch_predict_job",
        model=model_upload.outputs["model"],
        bigquery_source_input_uri=bq_eval_table,
        bigquery_destination_output_uri=f"bq://{project}",
        instances_format="bigquery",
        predictions_format="bigquery",
        model_parameters={},
        machine_type=DEPLOY_COMPUTE,
        starting_replica_count=min_replica_count,
        max_replica_count=max_replica_count,
        accelerator_type=accelerator_type,
        accelerator_count=accelerator_count,
    ).after(model_upload)

    batch_eval = ModelEvaluationOp(
        project=project,
        root_dir=bucket,
        problem_type="classification",
        classification_type="multiclass",
        ground_truth_column=label,
        class_names=class_names,
        predictions_format="jsonl",
        batch_prediction_job=batch_predict.outputs["batchpredictionjob"],
    )<jupyter_output><empty_output><jupyter_text>Compile and execute the BigQuery ML training, and batch model evaluation pipelineNext, you compile the pipeline and then execute it. The pipeline takes the following parameters, which are passed as the dictionary `parameter_values`:- `bq_train_table`: The BigQuery table containing the training data.- `bq_eval_table`: The BigQuery table containing the evaluation data.- `label`: The corresponding label for the BigQuery dataset.- `dataset`: The BigQuery dataset component name.- `model`: The BigQuery model component name.- `artifact_uri`: The Cloud Storage location to export the BigQuery model artifacts.- `num_trials`: If greater than one, will perform hyperparameter tuning for the specified number of trials using the Vertex AI Vizier service.- `deploy_image`: The container image for serving predictions.- `machine_type`: The VM for serving predictions.- `min_replica_count`/`max_replica_count`: The number of virtual machines for auto-scaling predictions.- `display_name`: Display name for Vertex AI Model resource.- `project`: The project ID.- `region`: The region.<jupyter_code>MODEL_DIR = BUCKET_NAME + "/bqmodel"

compiler.Compiler().compile(pipeline_func=pipeline, package_path="bqml.json")

pipeline = aip.PipelineJob(
    display_name="bqml",
    template_path="bqml.json",
    pipeline_root=PIPELINE_ROOT,
    parameter_values={
        "bq_train_table": BQ_TABLE_TRAIN,
        "bq_eval_table": IMPORT_EVAL,
        "label": "species",
        "class_names": [
            "Adelie Penguin (Pygoscelis adeliae)",
            "Chinstrap penguin (Pygoscelis antarctica)",
            "Gentoo penguin (Pygoscelis papua)",
        ],
        "dataset": "bqml_tutorial",
        "model": "penguins_model",
        "artifact_uri": MODEL_DIR,
        #'num_trials': 1,
        "deploy_image": DEPLOY_IMAGE,
        "display_name": "penguins",
        "machine_type": DEPLOY_COMPUTE,
        "min_replica_count": 1,
        "max_replica_count": 1,
        "accelerator_type": DEPLOY_GPU.name,
        "accelerator_count": 1,
        "bucket": BUCKET_NAME,
        "project": PROJECT_ID,
        "location": "US",
    },
    # enable_caching=False
)

pipeline.run()

! rm -rf bqml.json<jupyter_output><empty_output><jupyter_text>View the BigQuery ML training and batch evaluation pipeline results<jupyter_code>PROJECT_NUMBER = pipeline.gca_resource.name.split("/")[1]
print(PROJECT_NUMBER)


def print_pipeline_output(job, output_task_name):
    JOB_ID = job.name
    print(JOB_ID)
    for _ in range(len(job.gca_resource.job_detail.task_details)):
        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id
        EXECUTE_OUTPUT = (
            PIPELINE_ROOT
            + "/"
            + PROJECT_NUMBER
            + "/"
            + JOB_ID
            + "/"
            + output_task_name
            + "_"
            + str(TASK_ID)
            + "/executor_output.json"
        )
        GCP_RESOURCES = (
            PIPELINE_ROOT
            + "/"
            + PROJECT_NUMBER
            + "/"
            + JOB_ID
            + "/"
            + output_task_name
            + "_"
            + str(TASK_ID)
            + "/gcp_resources"
        )
        if tf.io.gfile.exists(EXECUTE_OUTPUT):
            ! gsutil cat $EXECUTE_OUTPUT
            break
        elif tf.io.gfile.exists(GCP_RESOURCES):
            ! gsutil cat $GCP_RESOURCES
            break

    return EXECUTE_OUTPUT


print("bigquery-query-job")
artifacts = print_pipeline_output(pipeline, "bigquery-query-job")
print("\n\n")
print("bigquery-create-model-job")
artifacts = print_pipeline_output(pipeline, "bigquery-create-model-job")
print("\n\n")
print("bigquery-evaluate-model-job")
artifacts = print_pipeline_output(pipeline, "bigquery-evaluate-model-job")
print("\n\n")
print("bigquery-export-model-job")
artifacts = print_pipeline_output(pipeline, "bigquery-export-model-job")
print("\n\n")
print("model-upload")
artifacts = print_pipeline_output(pipeline, "model-upload")
print("\n\n")
print("model-batch-predict")
artifacts = print_pipeline_output(pipeline, "model-batch-predict")
output = !gsutil cat $artifacts
output = json.loads(output[0])
print("\n\n")
print(
    output["artifacts"]["batchpredictionjob"]["artifacts"][0]["metadata"][
        "gcsOutputDirectory"
    ]
)
print("model-evaluation")
artifacts = print_pipeline_output(pipeline, "model-evaluation")<jupyter_output><empty_output><jupyter_text>Delete a pipeline jobAfter a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`.<jupyter_code>pipeline.delete()<jupyter_output><empty_output><jupyter_text>Delete the BigQuery model and datasetNext, delete the BigQuery model and dataset.<jupyter_code>try:
    job = bqclient.delete_model("bqml_tutorial.penguins_model")
except:
    pass
job = bqclient.delete_dataset("bqml_tutorial", delete_contents=True)<jupyter_output><empty_output><jupyter_text>Cleaning upTo clean up all Google Cloud resources used in this project, you can [delete the Google Cloudproject](https://cloud.google.com/resource-manager/docs/creating-managing-projectsshutting_down_projects) you used for the tutorial.Otherwise, you can delete the individual resources you created in this tutorial.- Dataset- Pipeline- Model- Endpoint- AutoML Training Job- Batch Job- Custom Job- Hyperparameter Tuning Job- Cloud Storage Bucket<jupyter_code>delete_all = True

if delete_all:
    # Delete the dataset using the Vertex dataset object
    try:
        if "dataset" in globals():
            dataset.delete()
    except Exception as e:
        print(e)

    # Delete the model using the Vertex model object
    try:
        if "model" in globals():
            model.delete()
    except Exception as e:
        print(e)

    # Delete the endpoint using the Vertex endpoint object
    try:
        if "endpoint" in globals():
            endpoint.undeploy_all()
            endpoint.delete()
    except Exception as e:
        print(e)

    # Delete the AutoML or Pipeline training job
    try:
        if "dag" in globals():
            dag.delete()
    except Exception as e:
        print(e)

    # Delete the custom training job
    try:
        if "job" in globals():
            job.delete()
    except Exception as e:
        print(e)

    # Delete the batch prediction job using the Vertex batch prediction object
    try:
        if "batch_predict_job" in globals():
            batch_predict_job.delete()
    except Exception as e:
        print(e)

    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object
    try:
        if "hpt_job" in globals():
            hpt_job.delete()
    except Exception as e:
        print(e)

    if "BUCKET_NAME" in globals():
        ! gsutil rm -r $BUCKET_NAME<jupyter_output><empty_output>