<jupyter_start><jupyter_text>Batch processing with Argo WorfklowsIn this notebook we will dive into how you can run batch processing with Argo Workflows and Seldon Core.Dependencies:* Seldon core installed as per the docs with an ingress* Minio running in your cluster to use as local (s3) object storage* Argo Workfklows installed in cluster (and argo CLI for commands)  Setup Install Seldon CoreUse the notebook to [set-up Seldon Core with Ambassador or Istio Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html).Note: If running with KIND you need to make sure do follow [these steps](https://github.com/argoproj/argo/issues/2376issuecomment-595593237) as workaround to the `/.../docker.sock` known issue. Set up Minio in your clusterUse the notebook to [set-up Minio in your cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html). Create rclone configurationIn this example, our workflow stages responsible for pulling / pushing data to in-cluster MinIO S3 storage will use `rclone` CLI.In order to configure the CLI we will create a following secret:<jupyter_code>%%writefile rclone-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: rclone-config-secret
type: Opaque
stringData:
  rclone.conf: |
    [cluster-minio]
    type = s3
    provider = minio
    env_auth = false
    access_key_id = minioadmin
    secret_access_key = minioadmin
    endpoint = http://minio.minio-system.svc.cluster.local:9000
!kubectl apply -n default -f rclone-config.yaml<jupyter_output>secret/rclone-config-secret created<jupyter_text>Install Argo WorkflowsYou can follow the instructions from the official [Argo Workflows Documentation](https://github.com/argoproj/argoquickstart).You also need to make sure that argo has permissions to create seldon deployments - for this you can create a role:<jupyter_code>%%writefile role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - "*"
- apiGroups:
  - "apps"
  resources:
  - deployments
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - "*"
- apiGroups:
  - machinelearning.seldon.io
  resources:
  - "*"
  verbs:
  - "*"
!!kubectl apply -n default -f role.yaml<jupyter_output><empty_output><jupyter_text>A service account:<jupyter_code>!kubectl create -n default serviceaccount workflow<jupyter_output>serviceaccount/workflow created<jupyter_text>And a binding<jupyter_code>!kubectl create rolebinding workflow -n default --role=workflow --serviceaccount=default:workflow<jupyter_output>rolebinding.rbac.authorization.k8s.io/workflow created<jupyter_text>Create some input for our modelWe will create a file that will contain the inputs that will be sent to our model<jupyter_code>mkdir -p assets/
import os
import random

random.seed(0)
with open("assets/input-data.txt", "w") as f:
    for _ in range(10000):
        data = [random.random() for _ in range(4)]
        data = "[[" + ", ".join(str(x) for x in data) + "]]\n"
        f.write(data)<jupyter_output><empty_output><jupyter_text>Check the contents of the file<jupyter_code>!wc -l assets/input-data.txt
!head assets/input-data.txt<jupyter_output>10000 assets/input-data.txt
[[0.8444218515250481, 0.7579544029403025, 0.420571580830845, 0.25891675029296335]]
[[0.5112747213686085, 0.4049341374504143, 0.7837985890347726, 0.30331272607892745]]
[[0.4765969541523558, 0.5833820394550312, 0.9081128851953352, 0.5046868558173903]]
[[0.28183784439970383, 0.7558042041572239, 0.6183689966753316, 0.25050634136244054]]
[[0.9097462559682401, 0.9827854760376531, 0.8102172359965896, 0.9021659504395827]]
[[0.3101475693193326, 0.7298317482601286, 0.8988382879679935, 0.6839839319154413]]
[[0.47214271545271336, 0.1007012080683658, 0.4341718354537837, 0.6108869734438016]]
[[0.9130110532378982, 0.9666063677707588, 0.47700977655271704, 0.8653099277716401]]
[[0.2604923103919594, 0.8050278270130223, 0.5486993038355893, 0.014041700164018955]]
[[0.7197046864039541, 0.39882354222426875, 0.824844977148233, 0.6681532012318508]]<jupyter_text>Upload the file to our minio<jupyter_code>!mc mb minio-seldon/data
!mc cp assets/input-data.txt minio-seldon/data/<jupyter_output>[m[32;1mBucket created successfully `minio-seldon/data`.[0m
...-data.txt:  820.96 KiB / 820.96 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 71.44 MiB/s 0s[0m[0m[m[32;1m<jupyter_text>Create Argo WorkflowIn order to create our argo workflow we have made it simple so you can leverage the power of the helm charts.Before we dive into the contents of the full helm chart, let's first give it a try with some of the settings.We will run a batch job that will set up a Seldon Deployment with 10 replicas and 100 batch client workers to send requests.<jupyter_code>!helm template seldon-batch-workflow helm-charts/seldon-batch-workflow/ \
    --set workflow.name=seldon-batch-process \
    --set seldonDeployment.name=sklearn \
    --set seldonDeployment.replicas=10 \
    --set seldonDeployment.serverWorkers=1 \
    --set seldonDeployment.serverThreads=10 \
    --set batchWorker.workers=100 \
    --set batchWorker.payloadType=ndarray \
    --set batchWorker.dataType=data \
    | argo submit --serviceaccount workflow -
!argo list -n default
!argo get -n default seldon-batch-process
!argo -n default logs seldon-batch-process<jupyter_output>[32mseldon-batch-process-3626514072: time="2021-01-15T11:44:57.620Z" level=info msg="Starting Workflow Executor" version=v2.12.3[0m
[32mseldon-batch-process-3626514072: time="2021-01-15T11:44:57.622Z" level=info msg="Creating a K8sAPI executor"[0m
[32mseldon-batch-process-3626514072: time="2021-01-15T11:44:57.622Z" level=info msg="Executor (version: v2.12.3, build_date: 2021-01-05T00:54:54Z) initialized (pod: default/seldon-batch-process-3626514072) with template:\n{\"name\":\"create-seldon-resource-template\",\"arguments\":{},\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"resource\":{\"action\":\"create\",\"manifest\":\"apiVersion: machinelearning.seldon.io/v1\\nkind: SeldonDeployment\\nmetadata:\\n  name: \\\"sklearn\\\"\\n  namespace: default\\n  ownerReferences:\\n  - apiVersion: argoproj.io/v1alpha1\\n    blockOwnerDeletion: true\\n    kind: Workflow\\n    name: \\\"seldon-batch-process\\\"\\n    uid: \\\"511f64a2-0699-42[...]<jupyter_text>Check output in object storeWe can now visualise the output that we obtained in the object store.First we can check that the file is present:<jupyter_code>import json
wf_arr = !argo get -n default seldon-batch-process -o json
wf = json.loads("".join(wf_arr))
WF_UID = wf["metadata"]["uid"]
print(f"Workflow UID is {WF_UID}")
!mc ls minio-seldon/data/output-data-"$WF_UID".txt<jupyter_output>[m[32m[2021-01-15 11:46:42 GMT] [0m[33m 3.4MiB [0moutput-data-511f64a2-0699-42eb-897a-c0a57b24072c.txt
[0m<jupyter_text>Now we can output the contents of the file created using the `mc head` command.<jupyter_code>!mc cp minio-seldon/data/output-data-"$WF_UID".txt assets/output-data.txt
!head assets/output-data.txt
!argo delete -n default seldon-batch-process<jupyter_output>Workflow 'seldon-batch-process' deleted
StarcoderdataJupyterStructuredCleanDedup