ModelOps/08_ModelOps_GIT_PIMA_Python_H2OAutoML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "262f8d25",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo - Python H2O AutoML using Git\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ed8b9a2-99c7-4013-9a8e-9e4e47b52e0a",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/git_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea796a98",
   "metadata": {},
   "source": [
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction:</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "This notebook will cover the Operationalization of the PIMA diabetes use case with H2O model format. <b>H2O</b> is an open source, distributed in-memory machine learning library with linear scalability. H2O supports the most widely used statistical & machine learning algorithms including gradient boosted machines, generalized linear models, deep learning and more. The most used capabilities in H2O are the <b>AutoML</b> capabilities that provides.<br>In this example, we will use the AutoML capabilities to generate a H2O xgboost and operationalize it through ModelOps in the same Model Catalog than other trained models based on other libraries.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e38e603c",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Steps in this Notebook</b></p>\n",
    "<ol style = 'font-size:16px;font-family:Arial'>\n",
    "<li>Configure the Environment </li>\n",
    " <li>Connect to Vantage</li>\n",
    " <li>Define Training function</li>\n",
    " <li>Define Evaluate function</li>\n",
    " <li>Define Scoring function</li>\n",
    " <li>Define Model Metadata </li>\n",
    " <li>Commit and Push to Git to let ModelOps manage</li>\n",
    " <li>ModelOps full lifecycle till deployment</li>\n",
    " <li>ModelOps Monitoring</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19aec107",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "012151e5",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9177cfc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradatamodelops==7.0.3 h2o==3.44.0.3 matplotlib==3.8.2 install-jdk==1.0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1e9d17a-201f-4a87-b2c5-2a54d14212e6",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2614ad3b",
   "metadata": {
    "tags": []
   },
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "658ab039",
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import *\n",
    "import os\n",
    "import getpass\n",
    "import logging\n",
    "import sys\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4100eb04",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>2. Connect to Vantage</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a837e73d-d31b-4924-ab4e-4b3ae2c88395",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b67ba9cb-2f64-483c-9499-de27806334fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0956ea0-f7c9-4144-ad8c-db377b2afebf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=08_ModelOps_GIT_PIMA_Python_H2OAutoML.ipynb;' UPDATE FOR SESSION; ''')\n",
    "\n",
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\"\n",
    "\n",
    "# set the path to the local project repository for this model demo\n",
    "model_local_path = '~/modelops-demo-models/model_definitions/pima_h2o_automl'\n",
    "res = os.system(f'mkdir -p {model_local_path}/model_modules')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0347e54d-84c5-4655-a525-a29c03c108fe",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ced883ea-1c9c-411e-b6ba-0ce69d67260b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "164f3630-d221-40d3-aec8-78c063c559a4",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2a0cef4-5b5f-42f0-9523-82c4453184cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Aoa_Byom_Models')\n",
    "    execute_sql(query) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f525eb0-b32a-44ca-9a05-be985ca024b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    "\n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query) \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c77272e6-80c0-43c7-9772-827409b9fbbc",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f52b1a6-3c05-47f8-bc55-8d9112067861",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b78c03dc",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>3. Define Training Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The training function takes the following shape </p>\n",
    "\n",
    "```python\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # your training code\n",
    "    \n",
    "    # export model artefacts\n",
    "    mojo = model.download_mojo(path=context.artifact_output_path, get_genmodel_jar=True)\n",
    "    new_mojo = os.path.join(os.path.abspath(os.getcwd()), context.artifact_output_path, \"model.h2o\")\n",
    "    if os.path.isfile(new_mojo):\n",
    "        print(\"The file already exists\")\n",
    "    else:\n",
    "        # Rename the file\n",
    "        os.rename(mojo, new_mojo)\n",
    "    \n",
    "    record_training_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12b64643",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/training.py\n",
    "from teradataml import DataFrame\n",
    "from aoa import (\n",
    "    record_training_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "import os\n",
    "import h2o\n",
    "from h2o.automl import H2OAutoML\n",
    "\n",
    "\n",
    "def check_java():\n",
    "    try:\n",
    "        print(os.environ['JAVA_HOME'])\n",
    "    except:\n",
    "        print ('Installing Java...')\n",
    "        import jdk\n",
    "        jdk.install('17', path='/usr/local/jdk')\n",
    "        os.environ['JAVA_HOME'] = '/usr/local/jdk/jdk-17.0.9+9'\n",
    "\n",
    "\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    \n",
    "    # read training dataset from Teradata and convert to pandas\n",
    "    check_java()\n",
    "    h2o.init()\n",
    "    train_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "    train_hdf = h2o.H2OFrame(train_df.to_pandas())\n",
    "\n",
    "    # convert target column to categorical\n",
    "    train_hdf[target_name] = train_hdf[target_name].asfactor()\n",
    "\n",
    "    print(\"Starting training...\")\n",
    "  \n",
    "    # Execute AutoML on training data\n",
    "    aml = H2OAutoML(max_models=context.hyperparams['max_models'], seed=context.hyperparams['seed'])\n",
    "    aml.train(x=feature_names, y=target_name, training_frame=train_hdf)\n",
    "\n",
    "    # Here we are getting the best GBM algorithm for demo purposes\n",
    "    model = aml.get_best_model(algorithm=\"gbm\")\n",
    "    if not model:\n",
    "        model = aml.leader\n",
    "\n",
    "    print(\"Finished training\")\n",
    "\n",
    "    # export model artefacts\n",
    "    mojo = model.download_mojo(path=context.artifact_output_path, get_genmodel_jar=True)\n",
    "    new_mojo = os.path.join(os.path.abspath(os.getcwd()), context.artifact_output_path, \"model.h2o\")\n",
    "    if os.path.isfile(new_mojo):\n",
    "        print(\"The file already exists\")\n",
    "    else:\n",
    "        # Rename the file\n",
    "        os.rename(mojo, new_mojo)\n",
    "\n",
    "    print(\"Saved trained model\")\n",
    "\n",
    "    try:\n",
    "        model.varimp_plot(server=True, save_plot_path=os.path.join(os.path.abspath(os.getcwd()), context.artifact_output_path, \"feature_importance.png\"))\n",
    "        fi = model.varimp(True)\n",
    "        fix = fi[['variable','scaled_importance']]\n",
    "        fis = fix.to_dict('records')\n",
    "        feature_importance = {v['variable']:float(v['scaled_importance']) for (k,v) in enumerate(fis)}\n",
    "    except:\n",
    "        print(\"Warning: This model doesn't support feature importance (Stacked Ensemble)\")\n",
    "        aml.varimp_heatmap()\n",
    "        save_plot('Feature Heatmap', context=context)\n",
    "        feature_importance = {}\n",
    "\n",
    "    record_training_stats(train_df,\n",
    "                          features=feature_names,\n",
    "                          targets=[target_name],\n",
    "                          categorical=[target_name],\n",
    "                          feature_importance=feature_importance,\n",
    "                          context=context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bcc662b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# Check Java installation, installs if required\n",
    "try:\n",
    "    print(f\"Java is installed at {os.environ['JAVA_HOME']}\")\n",
    "except:\n",
    "    os.environ['JAVA_HOME'] = '/home/jovyan/.jdk/jdk-17.0.9+9'\n",
    "    if os.path.isdir(os.environ['JAVA_HOME']) is False:\n",
    "        print ('Installing Java...')\n",
    "        import jdk\n",
    "        jdk.install('17', path='/home/jovyan/.jdk')\n",
    "\n",
    "# define the training dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes\n",
    "FROM Demo_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN Demo_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 <> 0\n",
    "\"\"\"\n",
    "\n",
    "feature_metadata =  {\n",
    "    \"database\": \"demo_modelops\",\n",
    "    \"table\": \"aoa_statistics_metadata\"\n",
    "}\n",
    "hyperparams = {\"max_models\": 3, \"seed\": 1}\n",
    "\n",
    "entity_key = \"PatientId\"\n",
    "target_names = [\"HasDiabetes\"]\n",
    "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
    "\n",
    "from aoa import ModelContext, DatasetInfo\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"artifacts/\",\n",
    "                   model_version=\"h2oaml_v1\",\n",
    "                   model_table=\"model_h2oaml_v1\")\n",
    "\n",
    "sys.path.append(os.path.expanduser(f\"{model_local_path}/model_modules\"))\n",
    "import training\n",
    "training.train(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7180b306",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c94661e",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>4. Define Evaluation Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The evaluation function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    with open(f\"{context.artifact_input_path}/model.h2o\", \"rb\") as f:\n",
    "        model_bytes = f.read()\n",
    "    \n",
    "    model = store_byom_tmp(get_context(), \"byom_models_tmp\", context.model_version, model_bytes)\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_evaluation_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f2d678e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/evaluation.py\n",
    "from sklearn import metrics\n",
    "from teradataml import DataFrame, copy_to_sql, get_context, H2OPredict\n",
    "from aoa import (\n",
    "    record_evaluation_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    store_byom_tmp,\n",
    "    ModelContext\n",
    ")\n",
    "import json\n",
    "import os\n",
    "import matplotlib\n",
    "\n",
    "\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    with open(f\"{context.artifact_input_path}/model.h2o\", \"rb\") as f:\n",
    "        model_bytes = f.read()\n",
    "\n",
    "    model = store_byom_tmp(get_context(), \"byom_models_tmp\", context.model_version, model_bytes)\n",
    "\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "\n",
    "    byom_target_sql = \"CAST(prediction AS INT)\"\n",
    "\n",
    "    h2o = H2OPredict(\n",
    "        modeldata=model,\n",
    "        newdata=DataFrame.from_query(context.dataset_info.sql),\n",
    "        accumulate=[context.dataset_info.entity_key, target_name]\n",
    "    )\n",
    "\n",
    "    predictions_df = h2o.result\n",
    "\n",
    "    predictions_df.to_sql(table_name=\"predictions_tmp\", if_exists=\"replace\", temporary=True)\n",
    "\n",
    "    metrics_df = DataFrame.from_query(f\"\"\"\n",
    "    SELECT \n",
    "        {target_name} as y_test, \n",
    "        {byom_target_sql} as y_pred\n",
    "        FROM predictions_tmp\n",
    "    \"\"\")\n",
    "    metrics_df = metrics_df.to_pandas()\n",
    "\n",
    "    y_pred = metrics_df[[\"y_pred\"]]\n",
    "    y_test = metrics_df[[\"y_test\"]]\n",
    "\n",
    "    evaluation = {\n",
    "        'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),\n",
    "        'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),\n",
    "        'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),\n",
    "        'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))\n",
    "    }\n",
    "\n",
    "    with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
    "        json.dump(evaluation, f)\n",
    "\n",
    "    cf = metrics.confusion_matrix(y_test, y_pred)\n",
    "    display = metrics.ConfusionMatrixDisplay(confusion_matrix=cf)\n",
    "    display.plot()\n",
    "    save_plot('Confusion Matrix', context=context)\n",
    "\n",
    "    fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred)\n",
    "    roc_auc = metrics.auc(fpr, tpr)\n",
    "    display = metrics.RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=roc_auc, estimator_name=context.model_version)\n",
    "    display.plot()\n",
    "    save_plot('ROC Curve', context=context)\n",
    "\n",
    "    # calculate stats if training stats exist\n",
    "    if os.path.exists(f\"{context.artifact_input_path}/data_stats.json\"):\n",
    "        record_evaluation_stats(features_df=DataFrame.from_query(context.dataset_info.sql),\n",
    "                                predicted_df=DataFrame(\"predictions_tmp\"),\n",
    "                                context=context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c57c3fc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the evaluation dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM Demo_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN Demo_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"artifacts/\",\n",
    "                   artifact_input_path=\"artifacts/\",\n",
    "                   model_version=\"h2oaml_v1\",\n",
    "                   model_table=\"model_h2oaml_v1\")\n",
    "\n",
    "# drop volatile table from session if executing multiple times\n",
    "try:\n",
    "    db_drop_table('byom_models_tmp')\n",
    "except: \n",
    "    pass\n",
    "\n",
    "import evaluation\n",
    "evaluation.evaluate(context=ctx)\n",
    "\n",
    "# view evaluation results\n",
    "import json\n",
    "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
    "    print(json.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f05fd4d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b5c08a5",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>5. Define Scoring Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The scoring function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    with open(f\"{context.artifact_input_path}/model.h2o\", \"rb\") as f:\n",
    "        model_bytes = f.read()\n",
    "\n",
    "    model = store_byom_tmp(get_context(), \"byom_models_tmp\", context.model_version, model_bytes)\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_scoring_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8526111d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/scoring.py\n",
    "from teradataml import copy_to_sql, get_context, DataFrame, H2OPredict\n",
    "from aoa import (\n",
    "    record_scoring_stats,\n",
    "    aoa_create_context,\n",
    "    store_byom_tmp,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "\n",
    "def score(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "\n",
    "    with open(f\"{context.artifact_input_path}/model.h2o\", \"rb\") as f:\n",
    "        model_bytes = f.read()\n",
    "\n",
    "    model = store_byom_tmp(get_context(), \"byom_models_tmp\", context.model_version, model_bytes)\n",
    "\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    byom_target_sql = \"CAST(prediction AS INT)\"\n",
    "\n",
    "    print(\"Scoring\")\n",
    "    h2o = H2OPredict(\n",
    "        modeldata=model,\n",
    "        newdata=DataFrame.from_query(context.dataset_info.sql),\n",
    "        accumulate=context.dataset_info.entity_key)\n",
    "\n",
    "    print(\"Finished Scoring\")\n",
    "\n",
    "\n",
    "    # store the predictions\n",
    "    predictions_df = h2o.result\n",
    "    \n",
    "    # add job_id column so we know which execution this is from if appended to predictions table\n",
    "    predictions_df = predictions_df.assign(job_id=context.job_id)\n",
    "    cols = {}\n",
    "    cols[target_name] = predictions_df['prediction']\n",
    "    predictions_df = predictions_df.assign(**cols)\n",
    "    predictions_df = predictions_df[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
    "\n",
    "    copy_to_sql(df=predictions_df,\n",
    "                schema_name=context.dataset_info.predictions_database,\n",
    "                table_name=context.dataset_info.predictions_table,\n",
    "                index=False,\n",
    "                if_exists=\"append\")\n",
    "\n",
    "    print(\"Saved predictions in Teradata\")\n",
    "    \n",
    "    # calculate stats\n",
    "    predictions_df = DataFrame.from_query(f\"\"\"\n",
    "        SELECT \n",
    "            * \n",
    "        FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
    "            WHERE job_id = '{context.job_id}'\n",
    "    \"\"\")\n",
    "\n",
    "    record_scoring_stats(features_df=DataFrame.from_query(context.dataset_info.sql),\n",
    "                         predicted_df=predictions_df,\n",
    "                         context=context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5364936",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the scoring dataset \n",
    "\n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*\n",
    "FROM Demo_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "    WHERE F.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "# where to store predictions\n",
    "predictions = {\n",
    "    \"database\": \"demo_user\",\n",
    "    \"table\": \"pima_patient_predictions_tmp\"\n",
    "}\n",
    "\n",
    "import uuid\n",
    "job_id=str(uuid.uuid4())\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata,\n",
    "                           predictions=predictions)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"artifacts/\",\n",
    "                   artifact_input_path=\"artifacts/\",\n",
    "                   model_version=\"h2oaml_v1\",\n",
    "                   model_table=\"model_h2oaml_v1\",\n",
    "                   job_id=job_id)\n",
    "\n",
    "# drop volatile table from session if executing multiple times\n",
    "try:\n",
    "    db_drop_table('byom_models_tmp')\n",
    "except: \n",
    "    pass\n",
    "\n",
    "import scoring\n",
    "scoring.score(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1f24b26",
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(f\"SELECT * FROM Demo_User.pima_patient_predictions_tmp WHERE job_id='{job_id}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2eb7505",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up\n",
    "\n",
    "os.system('rm -f artifacts/*')\n",
    "\n",
    "try:\n",
    "    db_drop_table('model_h2oaml_v1')\n",
    "except: \n",
    "    pass\n",
    "\n",
    "try:\n",
    "    db_drop_table('pima_patient_predictions_tmp')\n",
    "except: \n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00c9e4a8",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>6. Define Model Metadata</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now let's create the configuration files.<br>Requirements file with the dependencies and versions:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb63c1e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/requirements.txt\n",
    "teradatamodelops==7.0.3\n",
    "teradataml==17.20.0.6\n",
    "pandas==2.1.4\n",
    "matplotlib==3.8.2\n",
    "scikit-learn==1.3.2\n",
    "h2o==3.44.0.3\n",
    "install-jdk==1.0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36cea7f3",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The hyper parameter configuration (default values):</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aa9e77b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/config.json\n",
    "{\n",
    "   \"hyperParameters\": {\n",
    "      \"max_models\": 10,\n",
    "      \"seed\": 1\n",
    "   }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d98bb820",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The model configuration:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4e1cb86",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model.json\n",
    "{\n",
    "    \"id\": \"cdce3e32-3f8c-417f-b10e-493f21740a92\",\n",
    "    \"name\": \"Python PIMA H2O AutoML\",\n",
    "    \"description\": \"Python PIMA H2O AutoML for Diabetes Prediction\",\n",
    "    \"language\": \"python\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfd08392",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>7. Commit and Push to Git to let ModelOps manage</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Run the command below to commit and push changes to our forked repository, so ModelOps can fetch the changes to the model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b96c64e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $model_local_path/../.. && git add . && git commit -m \"Added Python PIMA H2O AutoML demo model 💦\" && git push"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "252cc8d5",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now that changes are pushed, you can make the lifecycle inside <b>ModelOps User Interface</b>, plan for new trainings, evaluations, scorings. Compare models and operationalize into Production with automated Monitoring and alerting capabilities.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e1ede4",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. ModelOps full lifecycle till deployment</b></p> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab9295e",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Use or Create a Project with the git code repository with the model code, then you should see the model in the catalog already created</p>\n",
    "\n",
    "<img src=\"images/08_01.png\" alt=\"Model Catalog with inDB\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select the Model and then click Train a new Model. Use default hyper-parameters. This will launch the training job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_02.png\" alt=\"Train\"/>\n",
    "\n",
    "<img src=\"images/08_03.png\" alt=\"Train job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_04.png\" alt=\"Train finished\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When Model is trained a new Model Id is created and you can get inside the Model Lifecycle screen to review artifacts and other details</p>\n",
    "\n",
    "<img src=\"images/08_06.png\" alt=\"Model lifecycle\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, let's evaluate the Model, click the button and select the evaluation dataset. This will launch the evaluation job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_07.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/> <img src=\"images/08_08.png\" alt=\"Evaluation job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When evaluation job is finished a Model evaluation Report is generated with the metrics and charts that evaluation script generates</p>\n",
    "\n",
    "<img src=\"images/08_26.png\" alt=\"Model Report\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now, let's approve the model and provide an approval description</p>\n",
    "\n",
    "<img src=\"images/08_09.png\" alt=\"Approval\" />\n",
    "\n",
    "<img src=\"images/08_10.png\" alt=\"Approval description\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The model is ready to be deployed. Let's deploy using a Batch scheduling option - Run it manual</p>\n",
    "\n",
    "<img src=\"images/08_11.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_12.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_13.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "464d131f-1a18-4517-aa96-16ae2a4a12b0",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "266b0115-03cd-42f1-8287-4a78c093e333",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1975717c",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>9. ModelOps Monitoring</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now the model is deployed and a new Deployment appears in the deployment screen</p>\n",
    "\n",
    "\n",
    "<img src=\"images/08_15.png\" alt=\"Deploymet\" />\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can run jobs manually from here, review history of executions and view the predictions for a specific job</p>\n",
    "\n",
    "<img src=\"images/08_16.png\" alt=\"Deployment Run\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_17.png\" alt=\"Deployment Jobs\" />\n",
    "\n",
    "<img src=\"images/08_18.png\" alt=\"Deployment view\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<img src=\"images/08_19.png\" alt=\"Deployment predictions\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_20.png\" alt=\"Deployment\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Feature Drift and Prediction Drift tabs you can check on the monitoring of the data drift</p>\n",
    "\n",
    "<img src=\"images/08_22.png\" alt=\"Feature Drift\" />\n",
    "\n",
    "<img src=\"images/08_21.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "<img src=\"images/08_23.png\" alt=\"Performance Monitoring\" />\n",
    "\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the Performance Drift, you can review multiple evaluations, let's evaluate the model with a new dataset. We create a new evaluation dataset with this query:</p>\n",
    "    \n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0  \n",
    "```\n",
    "\n",
    "<img src=\"images/08_24.png\" alt=\"Evaluate\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>and now see the evolution of the metrics</p>\n",
    "\n",
    "<img src=\"images/08_25.png\" alt=\"Metrics monitoring\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "With ModelOps you can close the cycle and review make decisions when you need to replace yor model in production, For example, You could get alerting from Data Drift of Performance Drift and you can create multiple versions and compare them, select a champion and deploy new versions that replace existing in Production.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331a1c68-423c-4fdf-8c9e-2c5c20ebdc07",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7298a689-a1a6-47a0-8483-da3992cdb00a",
   "metadata": {},
   "source": [
    "[![image](images/launchModelOps.png)](/modelops)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e35706f-cee7-40db-9f92-2f637ced36a6",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial;color:#00233C'>10. Cleanup</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a886b4a3-3cf0-41ff-93b5-f0922b35f617",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e686ef-5ba3-4bce-b555-d31e3f32ea69",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d619741f-fcf1-47a2-93ef-3e38f95284b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f19c2904-da4c-4b4b-9859-5d6c9f8a30af",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92cd05b4-29b4-447b-a43d-06ccd7513a99",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f1607c4-3210-4edc-9559-31e30053ba34",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "493a5d80-b762-4656-9533-35f62ce2966e",
   "metadata": {},
   "source": [
    "[<< Back to GIT PIMA H2OAutoML](./08_ModelOps_GIT_PIMA_Python_H2OAutoML.ipynb) | [Continue to GIT PIMA XGBoost >>](./09_ModelOps_GIT_PIMA_Python_XGboost.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "784043ac",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}