ModelOps/04_ModelOps_BYOM_PIMA_ONNX.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo - ONNX with ModelOps BYOM no code\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/byom_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction:</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This Notebook will show you how to work with the Bring Your Own Model (BYOM) pattern and BYOM In-Vantage Scoring. This deployment pattern allows you to use whatever data science platform you want to perform model training and import back to Vantage to do a direct deployment, scoring. For understanding more about BYOM you can review official user documentation.<br>When you use BYOM mechanisms you can deploy models directly from IDE environment, or make a proper operationalization through ModelOps. With ModelOps, you get <b>full governance</b> around your models deployment and you can <b>leverage ModelOps automation</b> for Validation, Scoring and Monitoring with an intuitive user interface that let you audit at any time all the information from your models and provide dashboards to monitor and review alerts from your deployments.<br>\n",
    "This notebook will cover the Operationalization of the PIMA diabetes use case with ONNX standard BYOM format. <b>ONNX</b> (Open Neural Network Exchange) is a very efficient model format which was created by Microsoft and which adoption as a standard open format is increasingly rapidly. While the name suggests it is primarily related to neural networks, it can be used with most machine learning libraries and algorithms like sklearn.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>\n",
    "    Steps in this Notebook</b></p>\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    " <li>Configure the Environment</li>\n",
    " <li>Connect to Vantage</li>\n",
    " <li>Train a model and export to ONNX</li>\n",
    " <li>Import the ONNX into ModelOps</li>\n",
    " <li>Go through Lifecycle - Evaluation (Automated Model Report)</li>\n",
    " <li>Go through Lifecycle - Approve </li>\n",
    " <li>Go through Lifecycle - Deploy (Publish and Schedule)</li>\n",
    " <li>Go through Lifecycle - Monitor (Data Drift and Performance)</li>\n",
    " <li>Configure Monitoring alert threshold (Optional) </li>\n",
    " <li>On demand Scoring from SQL (Optional)</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradataml==17.20.0.6 teradatamodelops==7.0.3  nyoka==4.3.0  skl2onnx==1.11.2  onnxmltools==1.7.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import *\n",
    "from teradatasqlalchemy.types import *\n",
    "import os\n",
    "import pandas as pd\n",
    "import getpass\n",
    "import logging\n",
    "import sys\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "import numpy as np\n",
    "from skl2onnx import to_onnx\n",
    "from skl2onnx import convert_sklearn, to_onnx, update_registered_converter\n",
    "from skl2onnx.common.shape_calculator import (\n",
    "    calculate_linear_classifier_output_shapes,\n",
    "    calculate_linear_regressor_output_shapes)\n",
    "from onnxmltools.convert.xgboost.operator_converters.XGBoost import convert_xgboost\n",
    "from onnxmltools.convert import convert_xgboost as convert_xgboost_booster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>2. Connect to Vantage</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username = 'demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=04_ModelOps_BYOM_PIMA_ONNX;' UPDATE FOR SESSION;''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Aoa_Byom_Models')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>3. Train a model and export to ONNX</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In the previous notebook <strong><a href=\"./02_ModelOps_PIMA_Experimentation.ipynb\">Pima Experimentation</a></strong> we have done the exploratory data analysis. Now, we wil create prediction model on the Pima dataset. We are using python modules to create our model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_pdf = DataFrame.from_query(f\"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM Demo_Modelops.pima_patient_features F\n",
    "JOIN Demo_Modelops.pima_patient_diagnoses D\n",
    "    ON F.patientid = D.patientid \n",
    "    WHERE F.patientid MOD 5 <> 0\n",
    "\"\"\").to_pandas(all_rows=True)\n",
    "\n",
    "features = [\"NumTimesPrg\", \"Age\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\"]\n",
    "target = \"HasDiabetes\"\n",
    "\n",
    "# split data into X and y\n",
    "X_train = train_pdf[features]\n",
    "y_train = train_pdf[target]\n",
    "\n",
    "model = Pipeline([('scaler', MinMaxScaler()),\n",
    "                  ('xgb', XGBClassifier(eta=0.2, max_depth=6))])\n",
    "\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>3.1 Convert the model to ONNX</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We can also convert the model to onnx format. This is a bit more involved as the client libraries for converting from sklearn/xgboost to onnx are not yet as mature.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "update_registered_converter(\n",
    "    XGBClassifier, 'XGBoostXGBClassifier',\n",
    "    calculate_linear_classifier_output_shapes, convert_xgboost,\n",
    "    options={'nocl': [True, False], 'zipmap': [True, False, 'columns']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_onnx = to_onnx(model, X_train.astype(np.float32), target_opset={'': 15, 'ai.onnx.ml': 2})\n",
    "with open(\"artifacts/model.onnx\", \"wb\") as f:\n",
    "    f.write(model_onnx.SerializeToString())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>4. Import into ModelOps to Operationalize</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Note some of the images may contain PMML because the operationalization steps are the same. Go to the ModelOps UI and import this as a new model inside the <b>ModelOps Getting Started</b> project.<br>Navigate to the projects screen and select the <b>ModelOps Getting Started</b>project. Click on it to navigate to the models screen of the project.</p>\n",
    "\n",
    "<img src=\"images/04_01.png\" alt=\"Projects screen\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you haven't added a default connection, you will be prompted to add one.<br>Once there, click on the <b>DEFINE BYOM MODEL</b> button on the top right of the screen.</p>\n",
    "\n",
    "<img src=\"images/04_02.png\" alt=\"Models screen\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Ensure that <b>Enable Model Monitoring</b> is checked and classification as model type is selected. Then set the table the one you have been using (probably your username) and use as Prediction Expression</p>\n",
    "\n",
    "```sql\n",
    "CAST(CAST(json_report AS JSON).JSONExtractValue('$.output_label[0]') AS INT)\n",
    "```\n",
    "<img src=\"images/04_03.png\" alt=\"BYOM model import sidesheeet\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then click on <b>SAVE</b>, and the import job will start.</p>\n",
    "\n",
    "<img src=\"images/04_04.png\" alt=\"BYOM model import job log\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>5. Go through Lifecycle - Evaluation (Automated Model Report)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Open the imported ModelID Lifecycle Screen. Review imported artifacts and evaluate the model with default logic (No code needed)</p>\n",
    "<img src=\"images/04_05.png\" alt=\"BYOM model lifecycle\"/>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select Evaluation dataset and run the evaluation process.</p>\n",
    "                              \n",
    "<img src=\"images/04_06.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now you can review the Model evaluation report generated with default metrics and confusion matrix plot - this can be customized with a custom evaluation logic (later in this notebook)<br>Click on View Report from ModelID Lifecycle screen.</p>\n",
    "\n",
    "<img src=\"images/04_07.png\" alt=\"Model Evaluation Report\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>6. Go through Lifecycle - Approve </b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Open the imported ModelID Lifecycle Screen. Click on Approve button to move forward on the next stage.<br>Include description of approval and accept</p>\n",
    "\n",
    "<img src=\"images/04_08.png\" alt=\"Approval\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>7. Go through Lifecycle - Deploy (Publish and Schedule)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Open the imported ModelID Lifecycle Screen. Click on Deploy button to publish Model in Vantage.</p>\n",
    "<img src=\"images/04_09.png\" alt=\"Model Lifecycle screen\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now select In-Vantage engine and click next</p>\n",
    "\n",
    "<img src=\"images/04_10.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now select which Connection (here you could use a Service Connection to run in Production optionally), Database and table you want to publish your BYOM model. Use your user \"demo_user\" and the table \"aoa_byom_models\"</p>\n",
    "\n",
    "<img src=\"images/04_11.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now select if you want to schedule your model scoring, dataset template to gather where you want to store your predictions and run the deploy process</p>\n",
    "\n",
    "<img src=\"images/04_12.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. Go through Lifecycle - Monitor (Data Drift and Performance)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Open the Deployment and review details of the deployed model.</p>\n",
    "<img src=\"images/04_14.png\" alt=\"Deployment details\"/>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can run the Prediction job from here , click the button to run the prediction job</p>\n",
    "<img src=\"images/04_15.png\" alt=\"Deployment Scoring Job\" />\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can also review the predictions, you can take this query and run into your SQL IDE or in Notebook later.</p>\n",
    "<img src=\"images/04_16.png\" alt=\"Deployment Predictions\" width=\"500\" height=\"500\"/> <img src=\"images/04_17.png\" alt=\"Deployment Predictions query\" width=\"500\" height=\"500\"/>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Review your Feature Drift, Prediction Drift. Here ModelOps shows the comparison of the distribution of data between training and evaluation/scoring, every scoring this gets updated and different KPIs are calculated. The one we used by default for monitoring is the Population Stability Index (PSI).</p>\n",
    "\n",
    "<img src=\"images/04_18.png\" alt=\"Feature Drift\" />\n",
    "<img src=\"images/04_19.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>For Performance Monitoring, we track over time the metrics of the model. For generating new metrics we need to create a new evaluation dataset and run an evaluation job.</p>\n",
    "\n",
    "<img src=\"images/04_20.png\" alt=\"Performance Drift\" />\n",
    "\n",
    "   \n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This has been generated using this query for evaluation dataset target:</p>\n",
    "\n",
    "```sql\n",
    "SELECT * FROM Demo_ModelOps.pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0 \n",
    "```    \n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now the performance has changed:</p>\n",
    "\n",
    "<img src=\"images/04_21.png\" alt=\"Performance Drift change\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>9. Configure Monitoring alert threshold (Optional) </b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "We can update the Alerts configuration for the models.<br>First enable Alert from Model catalog</p>\n",
    "\n",
    "<img src=\"images/04_23.png\" alt=\"Alert Configuration\" /> \n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, you can change your default alerting mechanism.<br>Go to your Model, and find the Tab \"Alert Configuration\"</p>\n",
    "\n",
    "<img src=\"images/04_22.png\" alt=\"Alert Configuration\" /> \n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's change the PSI value to 0.01 instead 0.2 <br>Use Edit option:</p>\n",
    "\n",
    "<img src=\"images/04_24.png\" alt=\"Alert Configuration PSI\" width=\"500\" height=\"500\"/> <img src=\"images/04_25.png\" alt=\"Alert Configuration PSI\" width=\"500\" height=\"500\"/> \n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>After few minutes, this has generated new alerts on the model deployed\n",
    "Go to Menu -> Alert and review the alerts generated for your model</p>\n",
    "\n",
    "<img src=\"images/04_26.png\" alt=\"Alerts\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now go to ModelID and click on View Model Drift</p>\n",
    "\n",
    "<img src=\"images/04_27.png\" alt=\"Model Lifecycle View Model Drift\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now review the Feature drift, you can check this screen also from your deployments</p>\n",
    "<img src=\"images/04_28.png\" alt=\"Feature Drift\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>10. On demand Scoring from SQL (Optional)</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p><b style = 'font-size:18px;font-family:Arial;color:#00233C'>10.1 View Published Models</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "Once deployed via ModelOps, we can view the models published to vantage by querying the table they are published to. Note this information is available via the AOA APIs also.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(\"SELECT TOP 10 * FROM aoa_byom_models WHERE model_type='ONNX'\").head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:18px;font-family:Arial;color:#00233C'>10.2 On-Demand Scoring</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Configuring VAL and BYOM locations:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Replace model_version with the one we have operationalized:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#teradataml version\n",
    "model_version=\"7f050da5-59db-45b7-9ccb-c194dce99dd3\"\n",
    "\n",
    "model = DataFrame.from_query(f\"\"\"\n",
    "SELECT * FROM aoa_byom_models \n",
    "    WHERE model_version='{model_version}'\n",
    "\"\"\")\n",
    "\n",
    "\n",
    "preds = ONNXPredict(\n",
    "        modeldata=model,\n",
    "        newdata=DataFrame.from_query(\"SELECT * FROM Demo_ModelOps.pima_patient_features WHERE patientid MOD 5 = 0\"),\n",
    "        accumulate=['PatientId'])\n",
    "\n",
    "preds.result.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#SQL version\n",
    "query = f\"\"\"\n",
    "SELECT * FROM MLDB.ONNXPredict (\n",
    "    ON (SELECT * FROM Demo_ModelOps.pima_patient_features WHERE patientid MOD 5 = 0) AS InputTable\n",
    "    ON (SELECT * FROM aoa_byom_models \n",
    "            WHERE model_version='{model_version}') AS ModelTable DIMENSION\n",
    "    USING\n",
    "      Accumulate ('patientid')\n",
    ") AS td;\n",
    "\"\"\"\n",
    "\n",
    "DataFrame.from_query(query).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>11. Cleanup</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[<< Back to PIMA PMML BYOM](./03_ModelOps_BYOM_PIMA_PMML.ipynb) | [Continue to PIMA H2O BYOM >>](./05_ModelOps_BYOM_PIMA_H2O.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}