ModelOps/11_ModelOps_GIT_PIMA_Python_indb_XGboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc5ef2d4-514a-4c8a-8741-4336263ec7ec",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo: In-Database XGBoost using Git\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260bf2ce-6b27-468d-9200-1e928d8709c2",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/git_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e628a88",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p style ='font-size:18px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial;color:#00233C'>This Notebook will show you how to work with ClearScape Analytics in-database functions with ModelOps. With in-database analytics you can solve your scalable challenges by using Vantage to train and score your models. Whether you have a big volume of data or you want to avoid the data movement implementation to train models outside Vantage, you can use ModelOps to manage your Catalog of Models from multiple platforms including in-database algorithms.<br>To know more about in-database algorithms review teradata official documentation.</p>\n",
    " \n",
    "<p style='font-size:16px;font-family:Arial;color:#00233C'>This notebook will cover the Operationalization of the PIMA diabetes use case with Python using the Teradata In-database XGBoost model. <strong>XGBoost</strong> is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It is one of the most used libraries by the community that solve many data science problems in a fast and accurate way.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e89a61f-22f2-4ea0-8189-aee3e232a487",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Steps in this Notebook</b></p>\n",
    "\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>Configure the Environment </li>\n",
    "    <li>Connect to Vantage</li>\n",
    "    <li>Define Training function </li>\n",
    "    <li>Define Evaluate function </li>\n",
    "    <li>Define Scoring function</li>\n",
    "    <li>Define Model Metadata</li>\n",
    "    <li>Commit and Push to Git to let ModelOps manage</li>\n",
    "    <li>ModelOps full lifecycle till deployment</li>\n",
    "    <li>ModelOps Monitoring</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c675a412-ac1c-4dbb-b4e3-21e847c3295b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be62e431-e39a-4baa-b02d-1dc25a36f559",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b499f2f4-45a0-49ba-bdf8-7a786cf01e53",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradataml==17.20.0.6 teradatamodelops==7.0.3 matplotlib==3.8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5691c81c-09e7-4acc-ae80-f4007cfc4522",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b392c630-2106-4619-b4a8-ceaba6cb7ce2",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6fbde24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import (\n",
    "    create_context, \n",
    "    remove_context,\n",
    "    get_context,\n",
    "    get_connection,\n",
    "    DataFrame,\n",
    "    configure,\n",
    "    execute_sql\n",
    ")\n",
    "import os\n",
    "import getpass\n",
    "import logging\n",
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "245c1005-e835-430e-a48f-c3962b72f3c8",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>2. Connect to Vantage</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a1f46c-7798-4fc9-a710-750c56e8a299",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "702a57c1-6013-4f92-bbac-b4278f1fb86b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce03f507-ff4c-4d5d-902f-8c362fe59fa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=11_ModelOps_GIT_PIMA_Python_Indb_XGboost.ipynb;' UPDATE FOR SESSION; ''')\n",
    "\n",
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\"\n",
    "\n",
    "# set the path to the local project repository for this model demo\n",
    "model_local_path = '~/modelops-demo-models/model_definitions/pima_python_indb_xgboost'\n",
    "res = os.system(f'mkdir -p {model_local_path}/model_modules')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16dd3af9-17a6-40e8-9dec-d2023cbaed5a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c65dde8-c2e9-4bc6-b1f8-b331a8d6ae19",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5abc3f95-2694-4f13-af04-3de523a04430",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a3e8f5-2ee8-4ace-83b7-e56c083effab",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE DEMO_USER.Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    execute_sql('DROP TABLE DEMO_USER.Aoa_Byom_Models;')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d6c0b9-f6b9-416c-b788-11cd8a6a9723",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE DEMO_USER.Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    execute_sql('DROP TABLE DEMO_USER.Pima_Patient_Predictions;')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c1c700-ba57-4c31-9fd2-3434023e6640",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be4b98a7-ed3a-4873-89b9-538bfbdc51a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acfea1e5-d74c-4b41-88f3-b40ed80f0d9b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>3. Define Training Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The training function takes the following shape </p>\n",
    "\n",
    "```python\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # your training code using teradataml indDB function\n",
    "    model = <InDB Function>(...)\n",
    "    \n",
    "    # save your model\n",
    "    model.result.to_sql(f\"model_${context.model_version}\", if_exists=\"replace\")  \n",
    "    \n",
    "    record_training_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7fb724",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/training.py\n",
    "from teradataml import (\n",
    "    DataFrame,\n",
    "    XGBoost,\n",
    "    ScaleFit,\n",
    "    ScaleTransform,\n",
    ")\n",
    "\n",
    "from aoa import (\n",
    "    record_training_stats,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "from collections import Counter\n",
    "\n",
    "# Define a function to recursively traverse a decision tree and count the usage of features\n",
    "def traverse_tree(tree, feature_counter):\n",
    "    if 'split_' in tree and 'attr_' in tree['split_']:\n",
    "        feature_counter[tree['split_']['attr_']] += 1\n",
    "    if 'leftChild_' in tree:\n",
    "        traverse_tree(tree['leftChild_'], feature_counter)\n",
    "    if 'rightChild_' in tree:\n",
    "        traverse_tree(tree['rightChild_'], feature_counter)\n",
    "\n",
    "# Compute feature importance based on tree traversal\n",
    "def compute_feature_importance(trees_json):\n",
    "    feature_counter = Counter()\n",
    "    for tree_json in trees_json:\n",
    "        tree = json.loads(tree_json)\n",
    "        traverse_tree(tree, feature_counter)\n",
    "    total_splits = sum(feature_counter.values())\n",
    "    feature_importance = {feature: count / total_splits for feature, count in feature_counter.items()}\n",
    "    return feature_importance\n",
    "\n",
    "\n",
    "def plot_feature_importance(fi, img_filename):\n",
    "    feat_importances = pd.Series(fi)\n",
    "    feat_importances.nlargest(10).plot(kind='barh').set_title('Feature Importance')\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "    \n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # Extracting feature names, target name, and entity key from the context\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    # Load the training data from Teradata\n",
    "    train_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "\n",
    "    print (\"Scaling using InDB Functions...\")\n",
    "    \n",
    "    # Scale the training data using the ScaleFit and ScaleTransform functions\n",
    "    scaler = ScaleFit(\n",
    "        data=train_df,\n",
    "        target_columns = feature_names,\n",
    "        scale_method = context.hyperparams[\"scale_method\"],\n",
    "        miss_value = context.hyperparams[\"miss_value\"],\n",
    "        global_scale = context.hyperparams[\"global_scale\"].lower() in ['true', '1'],\n",
    "        multiplier = context.hyperparams[\"multiplier\"],\n",
    "        intercept = context.hyperparams[\"intercept\"]\n",
    "    )\n",
    "\n",
    "    scaled_train = ScaleTransform(\n",
    "        data=train_df,\n",
    "        object=scaler.output,\n",
    "        accumulate = [target_name,entity_key]\n",
    "    )\n",
    "    \n",
    "    scaler.output.to_sql(f\"scaler_${context.model_version}\", if_exists=\"replace\")\n",
    "    print(\"Saved scaler\")\n",
    "    \n",
    "    print(\"Starting training...\")\n",
    "\n",
    "    # Train the model using XGBoost\n",
    "    model = XGBoost(\n",
    "        data = scaled_train.result,\n",
    "        input_columns = feature_names,\n",
    "        response_column = target_name,\n",
    "        model_type = context.hyperparams[\"model_type\"],\n",
    "        lambda1 = context.hyperparams[\"lambda1\"],\n",
    "    )\n",
    "\n",
    "    # Save the trained model to SQL\n",
    "    model.result.to_sql(f\"model_${context.model_version}\", if_exists=\"replace\")  \n",
    "    print(\"Saved trained model\")\n",
    "\n",
    "    # Calculate feature importance and generate plot\n",
    "    model_pdf = model.result.to_pandas()['classification_tree']\n",
    "    feature_importance = compute_feature_importance(model_pdf)\n",
    "    plot_feature_importance(feature_importance, f\"{context.artifact_output_path}/feature_importance\")\n",
    "    \n",
    "\n",
    "    record_training_stats(\n",
    "        train_df,\n",
    "        features=feature_names,\n",
    "        targets=[target_name],\n",
    "        categorical=[target_name],\n",
    "        feature_importance=feature_importance,\n",
    "        context=context\n",
    "    )\n",
    "    \n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c437521",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the training dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes\n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN DEMO_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 <> 0\n",
    "\"\"\"\n",
    "\n",
    "feature_metadata =  {\n",
    "    \"database\": \"DEMO_ModelOps\",\n",
    "    \"table\": \"aoa_statistics_metadata\"\n",
    "}\n",
    "hyperparams = {\n",
    "    \"model_type\": \"Classification\",\n",
    "    \"scale_method\":\"RANGE\",\n",
    "    \"miss_value\":\"KEEP\",\n",
    "    \"global_scale\": \"False\",\n",
    "    \"multiplier\":\"1\",\n",
    "    \"intercept\":\"0\",\n",
    "    \"max_depth\": 8,\n",
    "    \"num_boosted_trees\": 100,\n",
    "    \"tree_size\": 0.5,\n",
    "    \"lambda1\" : 1.0\n",
    "}\n",
    "\n",
    "entity_key = \"PatientId\"\n",
    "target_names = [\"HasDiabetes\"]\n",
    "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
    " \n",
    "from aoa import ModelContext, DatasetInfo\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   model_version=\"indb_xgboost_v1\",\n",
    "                   model_table=\"model_indb_xgboost_v1\")\n",
    "\n",
    "sys.path.append(os.path.expanduser(f\"{model_local_path}/model_modules\"))\n",
    "import training\n",
    "training.train(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15cf5f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33498ab1-a54c-4694-950d-ad3d3b6fd207",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>4. Define Evaluation Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The evaluation function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model from Vantage\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_evaluation_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d751b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/evaluation.py\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "from teradataml import(\n",
    "    DataFrame, \n",
    "    copy_to_sql, \n",
    "    get_context, \n",
    "    get_connection, \n",
    "    ScaleTransform, \n",
    "    XGBoostPredict, \n",
    "    ConvertTo, \n",
    "    ClassificationEvaluator,\n",
    "    ROC\n",
    ")\n",
    "from aoa import (\n",
    "    record_evaluation_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import joblib\n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "\n",
    "# Define function to traverse a decision tree and count feature occurrences\n",
    "def traverse_tree(tree, feature_counter):\n",
    "    if 'split_' in tree and 'attr_' in tree['split_']:\n",
    "        feature_counter[tree['split_']['attr_']] += 1\n",
    "    if 'leftChild_' in tree:\n",
    "        traverse_tree(tree['leftChild_'], feature_counter)\n",
    "    if 'rightChild_' in tree:\n",
    "        traverse_tree(tree['rightChild_'], feature_counter)\n",
    "\n",
    "        \n",
    "# Define function to compute feature importance from tree structures\n",
    "def compute_feature_importance(trees_json):\n",
    "    feature_counter = Counter()\n",
    "    for tree_json in trees_json:\n",
    "        tree = json.loads(tree_json)\n",
    "        traverse_tree(tree, feature_counter)\n",
    "    total_splits = sum(feature_counter.values())\n",
    "    feature_importance = {feature: count / total_splits for feature, count in feature_counter.items()}\n",
    "    return feature_importance\n",
    "\n",
    "\n",
    "# Define a function to plot feature importances\n",
    "def plot_feature_importance(fi, img_filename):\n",
    "    import pandas as pd\n",
    "    import matplotlib.pyplot as plt\n",
    "    feat_importances = pd.Series(fi)\n",
    "    feat_importances.nlargest(10).plot(kind='barh').set_title('Feature Importance')\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "\n",
    "# Define function to plot a confusion matrix from given data\n",
    "def plot_confusion_matrix(cf, img_filename):\n",
    "    import matplotlib.pyplot as plt\n",
    "    fig, ax = plt.subplots(figsize=(7.5, 7.5))\n",
    "    ax.matshow(cf, cmap=plt.cm.Blues, alpha=0.3)\n",
    "    for i in range(cf.shape[0]):\n",
    "        for j in range(cf.shape[1]):\n",
    "            ax.text(x=j, y=i,s=cf[i, j], va='center', ha='center', size='xx-large')\n",
    "    ax.set_xlabel('Predicted labels');\n",
    "    ax.set_ylabel('True labels'); \n",
    "    ax.set_title('Confusion Matrix');\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "\n",
    "# Define function to plot ROC curve from ROC output data \n",
    "def plot_roc_curve(roc_out, img_filename):\n",
    "    import matplotlib.pyplot as plt\n",
    "    auc = roc_out.result.to_pandas().reset_index()['AUC'][0]\n",
    "    roc_results = roc_out.output_data.to_pandas()\n",
    "    plt.plot(roc_results['fpr'], roc_results['tpr'], color='darkorange', lw=2, label='ROC curve (AUC = %0.2f)' % 0.27)\n",
    "    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operating Characteristic (ROC) Curve')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    # Load the trained model from SQL\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    # Load the test data from Teradata\n",
    "    test_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "\n",
    "    # Scaling the test set\n",
    "    print (\"Loading scaler...\")\n",
    "    scaler = DataFrame(f\"scaler_${context.model_version}\")\n",
    "\n",
    "    scaled_test = ScaleTransform(\n",
    "        data=test_df,\n",
    "        object=scaler,\n",
    "        accumulate = [target_name,entity_key]\n",
    "    )\n",
    "    \n",
    "    print(\"Evaluating...\")\n",
    "    # Make predictions using the XGBoostPredict function\n",
    "    predictions = XGBoostPredict(\n",
    "        object=model,\n",
    "        newdata=scaled_test.result,\n",
    "        model_type = 'Classification',\n",
    "        accumulate=target_name,\n",
    "        id_column=entity_key,\n",
    "        output_prob=True,\n",
    "        output_responses=['0','1'],\n",
    "        object_order_column=['task_index', 'tree_num', 'iter', 'class_num', 'tree_order']\n",
    "    )\n",
    "\n",
    "    # Convert the predicted data into the specified format\n",
    "    predicted_data = ConvertTo(\n",
    "        data = predictions.result,\n",
    "        target_columns = [target_name,'Prediction'],\n",
    "        target_datatype = [\"INTEGER\"]\n",
    "    )\n",
    "\n",
    "    # Evaluate classification metrics using ClassificationEvaluator\n",
    "    ClassificationEvaluator_obj = ClassificationEvaluator(\n",
    "        data=predicted_data.result,\n",
    "        observation_column=target_name,\n",
    "        prediction_column='Prediction',\n",
    "        num_labels=2\n",
    "    )\n",
    "\n",
    "     # Extract and store evaluation metrics\n",
    "    metrics_pd = ClassificationEvaluator_obj.output_data.to_pandas()\n",
    "\n",
    "    evaluation = {\n",
    "        'Accuracy': '{:.2f}'.format(metrics_pd.MetricValue[0]),\n",
    "        'Micro-Precision': '{:.2f}'.format(metrics_pd.MetricValue[1]),\n",
    "        'Micro-Recall': '{:.2f}'.format(metrics_pd.MetricValue[2]),\n",
    "        'Micro-F1': '{:.2f}'.format(metrics_pd.MetricValue[3]),\n",
    "        'Macro-Precision': '{:.2f}'.format(metrics_pd.MetricValue[4]),\n",
    "        'Macro-Recall': '{:.2f}'.format(metrics_pd.MetricValue[5]),\n",
    "        'Macro-F1': '{:.2f}'.format(metrics_pd.MetricValue[6]),\n",
    "        'Weighted-Precision': '{:.2f}'.format(metrics_pd.MetricValue[7]),\n",
    "        'Weighted-Recall': '{:.2f}'.format(metrics_pd.MetricValue[8]),\n",
    "        'Weighted-F1': '{:.2f}'.format(metrics_pd.MetricValue[9]),\n",
    "    }\n",
    "\n",
    "     # Save evaluation metrics to a JSON file\n",
    "    with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
    "        json.dump(evaluation, f)\n",
    "        \n",
    "    # Generate and save confusion matrix plot\n",
    "    cm = confusion_matrix(predicted_data.result.to_pandas()['HasDiabetes'], predicted_data.result.to_pandas()['Prediction'])\n",
    "    plot_confusion_matrix(cm, f\"{context.artifact_output_path}/confusion_matrix\")\n",
    "\n",
    "    # Generate and save ROC curve plot\n",
    "    roc_out = ROC(\n",
    "        data=predictions.result,\n",
    "        probability_column='Prob_1',\n",
    "        observation_column=target_name,\n",
    "        positive_class='1',\n",
    "        num_thresholds=1000\n",
    "    )\n",
    "    plot_roc_curve(roc_out, f\"{context.artifact_output_path}/roc_curve\")\n",
    "\n",
    "    # Calculate feature importance and generate plot\n",
    "    try:\n",
    "        model_pdf = model.result.to_pandas()['classification_tree']\n",
    "        feature_importance = compute_feature_importance(model_pdf)\n",
    "        feature_importance_df = pd.DataFrame(list(feature_importance.items()), columns=['Feature', 'Importance'])\n",
    "        plot_feature_importance(feature_importance, f\"{context.artifact_output_path}/feature_importance\")\n",
    "    except:\n",
    "        feature_importance = {}\n",
    "\n",
    "    predictions_table = \"predictions_tmp\"\n",
    "    copy_to_sql(df=predicted_data.result, table_name=predictions_table, index=False, if_exists=\"replace\", temporary=True)\n",
    "\n",
    "    # calculate stats if training stats exist\n",
    "    if os.path.exists(f\"{context.artifact_input_path}/data_stats.json\"):\n",
    "        record_evaluation_stats(\n",
    "            features_df=test_df,\n",
    "            predicted_df=DataFrame.from_query(f\"SELECT * FROM {predictions_table}\"),\n",
    "            feature_importance=feature_importance,\n",
    "            context=context\n",
    "        )\n",
    "\n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12ba7a9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the evaluation dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN DEMO_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   artifact_input_path=\"./artifacts\",\n",
    "                   model_version=\"indb_xgboost_v1\",\n",
    "                   model_table=\"model_indb_xgboost_v1\")\n",
    "\n",
    "import evaluation\n",
    "evaluation.evaluate(context=ctx)\n",
    "\n",
    "# view evaluation results\n",
    "import json\n",
    "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
    "    print(json.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ce4ba2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ae441da-9858-4a91-8f70-d51b92e0ee90",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>5. Define Scoring Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The scoring function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_scoring_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb4d4c1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/scoring.py\n",
    "from teradataml import (\n",
    "    copy_to_sql,\n",
    "    DataFrame,\n",
    "    XGBoostPredict,\n",
    "    ScaleTransform\n",
    ")\n",
    "from aoa import (\n",
    "    record_scoring_stats,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    \n",
    "    aoa_create_context()\n",
    "\n",
    "    # Load the trained model from SQL\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "\n",
    "    # Extract feature names, target name, and entity key from the context\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    # Load the test dataset\n",
    "    test_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "    features_tdf = DataFrame.from_query(context.dataset_info.sql)\n",
    "    features_pdf = features_tdf.to_pandas(all_rows=True)\n",
    "\n",
    "    # Scaling the test set\n",
    "    print (\"Loading scaler...\")\n",
    "    scaler = DataFrame(f\"scaler_${context.model_version}\")\n",
    "\n",
    "    # Scale the test dataset using the trained scaler\n",
    "    scaled_test = ScaleTransform(\n",
    "        data=test_df,\n",
    "        object=scaler,\n",
    "        accumulate = entity_key\n",
    "    )\n",
    "    \n",
    "    print(\"Scoring...\")\n",
    "    # Make predictions using the XGBoostPredict function\n",
    "    predictions = XGBoostPredict(\n",
    "        object=model,\n",
    "        newdata=scaled_test.result,\n",
    "        model_type = 'Classification',\n",
    "        id_column=entity_key,\n",
    "        output_prob=True,\n",
    "        output_responses=['0','1'],\n",
    "        object_order_column=['task_index', 'tree_num', 'iter', 'class_num', 'tree_order']\n",
    "    )\n",
    "     \n",
    "    # Convert predictions to pandas DataFrame and process\n",
    "    predictions_pdf = predictions.result.to_pandas(all_rows=True).rename(columns={\"Prediction\": target_name}).astype(int)\n",
    "\n",
    "    print(\"Finished Scoring\")\n",
    "\n",
    "    # store the predictions\n",
    "    predictions_pdf = pd.DataFrame(predictions_pdf, columns=[target_name])\n",
    "    predictions_pdf[entity_key] = features_pdf.index.values\n",
    "    # add job_id column so we know which execution this is from if appended to predictions table\n",
    "    predictions_pdf[\"job_id\"] = context.job_id\n",
    "\n",
    "    # teradataml doesn't match column names on append.. and so to match / use same table schema as for byom predict\n",
    "    # example (see README.md), we must add empty json_report column and change column order manually (v17.0.0.4)\n",
    "    # CREATE MULTISET TABLE pima_patient_predictions\n",
    "    # (\n",
    "    #     job_id VARCHAR(255), -- comes from airflow on job execution\n",
    "    #     PatientId BIGINT,    -- entity key as it is in the source data\n",
    "    #     HasDiabetes BIGINT,   -- if model automatically extracts target\n",
    "    #     json_report CLOB(1048544000) CHARACTER SET UNICODE  -- output of\n",
    "    # )\n",
    "    # PRIMARY INDEX ( job_id );\n",
    "    predictions_pdf[\"json_report\"] = \"\"\n",
    "    predictions_pdf = predictions_pdf[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
    "\n",
    "    copy_to_sql(\n",
    "        df=predictions_pdf,\n",
    "        schema_name=context.dataset_info.predictions_database,\n",
    "        table_name=context.dataset_info.predictions_table,\n",
    "        index=False,\n",
    "        if_exists=\"append\"\n",
    "    )\n",
    "    \n",
    "    print(\"Saved predictions in Teradata\")\n",
    "\n",
    "    # calculate stats\n",
    "    predictions_df = DataFrame.from_query(f\"\"\"\n",
    "        SELECT \n",
    "            * \n",
    "        FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
    "            WHERE job_id = '{context.job_id}'\n",
    "    \"\"\")\n",
    "\n",
    "    record_scoring_stats(features_df=features_tdf, predicted_df=predictions_df, context=context)\n",
    "\n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0edf735d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the scoring dataset \n",
    "\n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*\n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "    WHERE F.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "# where to store predictions\n",
    "predictions = {\n",
    "    \"database\": \"demo_user\",\n",
    "    \"table\": \"pima_patient_predictions_tmp\"\n",
    "}\n",
    "\n",
    "import uuid\n",
    "job_id=str(uuid.uuid4())\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata,\n",
    "                           predictions=predictions)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   artifact_input_path=\"./artifacts\",\n",
    "                   model_version=\"indb_xgboost_v1\",\n",
    "                   model_table=\"model_indb_xgboost_v1\",\n",
    "                   job_id=job_id)\n",
    "\n",
    "import scoring\n",
    "scoring.score(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e27b2370",
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(f\"SELECT * FROM pima_patient_predictions_tmp WHERE job_id='{job_id}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06db76e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up\n",
    "\n",
    "os.system('rm -f artifacts/*')\n",
    "\n",
    "try:\n",
    "    get_context().execute(f\"DROP TABLE model_indb_xgboost_v1\")\n",
    "except: \n",
    "    pass\n",
    "\n",
    "try:\n",
    "    get_context().execute(f\"DROP TABLE pima_patient_predictions_tmp\")\n",
    "except: \n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b5019e9-654f-4260-8a95-848539f709ba",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>6. Define Model Metadata</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now let's create the configuration files.<br>Requirements file with the dependencies and versions:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "157dafbf-a633-43f6-94d7-68c0e89668b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/requirements.txt\n",
    "pandas==2.1.4\n",
    "teradatamodelops==7.0.3\n",
    "matplotlib==3.8.2\n",
    "PyYAML==5.4.1\n",
    "scikit-learn==1.3.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f9f7d1",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>The hyper parameter configuration (default values):</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8711bb8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/config.json\n",
    "{\n",
    "   \"hyperParameters\": {\n",
    "    \"model_type\": \"Classification\",\n",
    "    \"scale_method\":\"RANGE\",\n",
    "    \"miss_value\":\"KEEP\",\n",
    "    \"global_scale\": \"False\",\n",
    "    \"multiplier\":\"1\",\n",
    "    \"intercept\":\"0\",\n",
    "    \"max_depth\": 8,\n",
    "    \"num_boosted_trees\": 100,\n",
    "    \"tree_size\": 0.5,\n",
    "    \"lambda1\" : 1.5\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ed3d03",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>The model configuration:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "224d5f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model.json\n",
    "{\n",
    "    \"id\": \"5b41d4d4-7236-54ab-846a-01c3151e1fd9\",\n",
    "    \"name\": \"In-database Diabetes Prediction XGBoost\",\n",
    "    \"description\": \"In-database XGBoost for Diabetes Prediction\",\n",
    "    \"language\": \"python\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "336a5344-9860-4835-a156-19547b5f6930",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>7. Commit and Push to Git to let ModelOps manage</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Run the command below to commit and push changes to our forked repository, so ModelOps can fetch the changes to the model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36decf79",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $model_local_path/../.. && git add . && git commit -m \"Added PIMA XGboost in database demo model \" && git push"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c7d023",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>Now that changes are pushed, you can make the lifecycle inside <strong>ModelOps User Interface</strong>, plan for new trainings, evaluations, scorings. Compare models and operationalize into Production with automated Monitoring and alerting capabilities.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c9013c-31bb-4324-a1b4-1cb6065b4294",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. ModelOps full lifecycle till deployment</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a34b63",
   "metadata": {},
   "source": [
    "<p style='font-size:16px;font-family:Arial'>Use or Create a Project with the git code repository with the model code, then you should see the model in the catalog already created</p>\n",
    "\n",
    "<img src=\"images/08_01.png\" alt=\"Model Catalog with inDB\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>Select the Model and then click Train a new Model. Use default hyper-parameters. This will launch the training job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_02.png\" alt=\"Train\"/>\n",
    "\n",
    "<img src=\"images/08_03.png\" alt=\"Train job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_04.png\" alt=\"Train finished\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>When Model is trained a new Model Id is created and you can get inside the Model Lifecycle screen to review artifacts and other details</p>\n",
    "\n",
    "<img src=\"images/08_06.png\" alt=\"Model lifecycle\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>Now, let's evaluate the Model, click the button and select the evaluation dataset. This will launch the evaluation job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_07.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/> <img src=\"images/08_08.png\" alt=\"Evaluation job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>When evaluation job is finished a Model evaluation Report is generated with the metrics and charts that evaluation script generates</p>\n",
    "\n",
    "<img src=\"images/08_26.png\" alt=\"Model Report\" />\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>Now, let's approve the model and provide an approval description</p>\n",
    "\n",
    "<img src=\"images/08_09.png\" alt=\"Approval\" />\n",
    "\n",
    "<img src=\"images/08_10.png\" alt=\"Approval description\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>The model is ready to be deployed. Let's deploy using a Batch scheduling option - Run it manual</p>\n",
    "\n",
    "<img src=\"images/08_11.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_12.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_13.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe60441-f62e-46c1-8baa-ff3cb0f1b809",
   "metadata": {},
   "source": [
    "<p style='font-size:16px;font-family:Arial'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b69248e-606c-4e46-99ea-dbed58ed3935",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>9. ModelOps Monitoring</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now the model is deployed and a new Deployment appears in the deployment screen</p>\n",
    "\n",
    "\n",
    "<img src=\"images/08_15.png\" alt=\"Deploymet\" />\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can run jobs manually from here, review history of executions and view the predictions for a specific job</p>\n",
    "\n",
    "<img src=\"images/08_16.png\" alt=\"Deployment Run\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_17.png\" alt=\"Deployment Jobs\" />\n",
    "\n",
    "<img src=\"images/08_18.png\" alt=\"Deployment view\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<img src=\"images/08_19.png\" alt=\"Deployment predictions\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_20.png\" alt=\"Deployment\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Feature Drift and Prediction Drift tabs you can check on the monitoring of the data drift</p>\n",
    "\n",
    "<img src=\"images/08_22.png\" alt=\"Feature Drift\" />\n",
    "\n",
    "<img src=\"images/08_21.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "<img src=\"images/08_23.png\" alt=\"Performance Monitoring\" />\n",
    "\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Performance Drift, you can review multiple evaluations, let's evaluate the model with a new dataset. We create a new evaluation dataset with this query:</p>\n",
    "    \n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0  \n",
    "```\n",
    "\n",
    "<img src=\"images/08_24.png\" alt=\"Evaluate\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>and now see the evolution of the metrics</p>\n",
    "\n",
    "<img src=\"images/08_25.png\" alt=\"Metrics monitoring\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "With ModelOps you can close the cycle and review make decisions when you need to replace yor model in production, For example, You could get alerting from Data Drift of Performance Drift and you can create multiple versions and compare them, select a champion and deploy new versions that replace existing in Production.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97beb6b4-ce0a-4787-bb8f-1653b9794dcd",
   "metadata": {},
   "source": [
    "<p style='font-size:16px;font-family:Arial'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9bd6e3e-d0b4-43ff-a4d3-b8a369c0a19c",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>10. Cleanup</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d1318f-fe9e-4c73-9428-9734ec1fbb8f",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a71990bc-09a9-4e58-bd20-5969c83523c0",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "420dd53e-f099-4f1b-a3f9-1757b47a4dda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b6f7b6-d739-4035-aef1-555f3bbf46c6",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0377bf79-cd6c-41c7-8684-84e7eb432b1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f1e45aa-131d-45c8-b494-7014d32a988b",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc01570c-13b8-4ba3-b07a-2245959a5a51",
   "metadata": {},
   "source": [
    "[<< Back to Git PIMA R GBM](./10_ModelOps_GIT_PIMA_R_GBM.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b46c6d-eab2-4fa4-9e40-2d9064305c46",
   "metadata": {},
   "source": [
    "<footer style=\"padding:10px;background:#f9f9f9;border-bottom:3px solid #394851\">Copyright © Teradata Corporation - 2023. All Rights Reserved.</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}