ModelOps/06_ModelOps_GIT_Project_Setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "28119940",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo: GIT, Project and Dataset Setup\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1d31cc-fec2-4991-9b7e-2f00f8eb7a28",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/project_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1497490b",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This Notebook will guide you to setup your own fork of the DEMO git repository, to create a new Project in ModelOps associated with your forked repository, and to create a Dataset.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In ClearScape Analytics Experience, we have pre-loaded a Project based on our official GIT repository. Now, you will create your own:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874bc0cd",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Steps in this Notebook</b></p>\n",
    "\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>Configure the Environment </li>\n",
    "    <li>Connect to Vantage</li>\n",
    "    <li>Setup a local repository (Optional)</li>\n",
    "    <li>Create a new Project in ModelOps</li>\n",
    "    <li>Create a Connection in ModelOps</li>\n",
    "    <li>Health Check the connection (Optional) </li>\n",
    "    <li>Create Dataset in ModelOps</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded3c17a-5d41-4f39-9782-685c31411daf",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2ca72a3-9848-4848-9364-e62ab151c47a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.1 Libraries installation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>A restart of the Kernel is needed to confirm changes</b>. We use -q parameter for a non-verbose log of the installation command, you may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2cafbc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install -q teradatamodelops==7.0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e310ba64-a6e3-492f-b1ac-e8cf2b913885",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f594ce7",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08d07ecd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from aoa import create_features_stats_table\n",
    "\n",
    "import getpass\n",
    "import logging\n",
    "import sys\n",
    "\n",
    "from teradataml import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aacdb35a-d96c-40ca-bf52-4db9844c6bdf",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>2. Connect to Vantage</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76fb2dfd-93ce-42fe-acd3-108e2b44af5a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6e0109b-84b6-4236-abc6-0e1bdfba1599",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9161d15c-190b-4ba6-901a-56a6efd1b0e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=06_ModelOps_GIT_Project_Setup.ipynb;' UPDATE FOR SESSION; ''')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c9e9efb-594d-4429-a2b9-05cba4639872",
   "metadata": {},
   "source": [
    "<h2 style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Getting Data for This Demo</b></h2>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d13c234-5172-49a6-8f13-f7e67ea86b22",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce98f9c5-5e30-4d2f-9371-5ba7d76062a8",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a3e8f5-2ee8-4ace-83b7-e56c083effab",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE Aoa_Byom_Models \n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Aoa_Byom_Models')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d6c0b9-f6b9-416c-b788-11cd8a6a9723",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Pima_Patient_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions \n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    "\n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c627567a-a5d6-47d8-8373-63b146836301",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be4b98a7-ed3a-4873-89b9-538bfbdc51a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7494a894",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>3. Setup local repository (Optional)</b>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>These steps are optional, but to understand the end-to-end experience of a new GIT model we share here the standard way to fork and clone the official demo repository.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e84d44b",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>3.1 Fork ModelOps demo models repository</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>First fork the repository <a href=\"https://github.com/Teradata/modelops-demo-models\">https://github.com/Teradata/modelops-demo-models</a> into your own account.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Open the link and click on the <strong>Fork</strong> button.</p>\n",
    "\n",
    "<img src=\"images/06_02A.png\" alt=\"GitHub Demo Models\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then fill in the required fields and select your personal account as the owner of the forked repository. Click on the <strong>Create fork</strong> button.</p>\n",
    "\n",
    "<img src=\"images/06_02B.png\" alt=\"Fork Demo Models\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>When done, copy the repo URL as you will need it in the next steps.</p>\n",
    "\n",
    "<img src=\"images/06_02C.png\" alt=\"Copy Demo Models URL\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "696dc63e",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>3.2 Generate a GitHub Personal Access Token</b>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From the previous screen, click on your account avatar to display the dropdown menu, and navigate to <strong>Settings</strong> (at the bottom).</p>\n",
    "\n",
    "<img src=\"images/06_02D.png\" alt=\"GitHub Profile Settings\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Scroll down the page to the bottom of the left menu. Then click on <strong>Developer settings</strong>.</p>\n",
    "\n",
    "<img src=\"images/06_02E.png\" alt=\"GitHub Developer Settings\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In that screen, open the <strong>Personal Access Token</strong> options on the left and click on <strong>(1) Tokens (classic)</strong>. Continue by clicking on <strong>(2) Generate new token</strong> and finally click on <strong>(3) Generate new token (Classic)</strong>.</p>\n",
    "\n",
    "<img src=\"images/06_02F.png\" alt=\"Personal Access Token\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Fill the fields as in this screenshot, and copy the new token (store it somewhere in your computer as you will need it later).</p>\n",
    "\n",
    "<img src=\"images/06_02G.png\" alt=\"Generate new token\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f38a251",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>3.3 Clone repo and setup git configuration</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Run the code blocks below, and provide the required info obtained in previous steps to finish cloning the repository locally and git configuration.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8cedd0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean repo in case we have previous clone\n",
    "# NOTE: this will remove all your git credentials stored previously\n",
    "!rm -fr ~/modelops-demo-models ~/.git-credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40ab4182",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "repo_url = input(\"Repository URL (paste here what you just copied):\")\n",
    "git_username = input(\"GIT Username:\")\n",
    "git_pat = getpass.getpass(\"GIT Personal Access Token:\")\n",
    "git_email = input(\"Your Email (the one you used to register at GitHub):\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd0ec8db",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone $repo_url ~/modelops-demo-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fa90a73",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git config --global user.name $git_username && git config --global user.email $git_email && git config --global credential.helper store && echo https://$git_username:$git_pat@github.com >> ~/.git-credentials && chmod 700 ~/.git-credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412f4202-6179-4b96-a8eb-7d4ae0e9c4bd",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Create a new Project</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This step is optional if you plan to use the DEMO project which is created by default in ModelOps installations.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>A project is used in ModelOps to organize your Model Catalog with your model versions. It is linked to a git repository, and can be assigned to a user or a group of users for collaboration.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Login into ModelOps and navigate to the Projects screen.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af901ed3-8a61-4b11-8c7e-4c68316cf23f",
   "metadata": {},
   "source": [
    "<img src=\"images/06_03.png\" alt=\"Projects\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "945d98fb-0cce-4aef-9391-7bd99e729fb1",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Click on the button <strong>CREATE PROJECT</strong> at the top-right of the screen.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f00f514b-3e0a-441f-b363-0b0842126fd7",
   "metadata": {},
   "source": [
    "<img src=\"images/06_04.png\" alt=\"Create Project\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e8a0616-b2da-4653-acec-5cde180e94e0",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Include:</p>\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li><strong>Name of the project:</strong> \"ModelOps Demo\"</li>\n",
    "    <li><strong>Description:</strong> \"Teradata ModelOps Demo Project\"</li>\n",
    "    <li><strong>Group:</strong> your user</li>\n",
    "    <li><strong>Path:</strong> your forked repository url</li>\n",
    "    <li><strong>Credentials:</strong> \"No Credentials\"</li>\n",
    "    <li><strong>Branch:</strong> \"master\"</li>\n",
    "</ul>\n",
    "    \n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can test connection with GIT with button <strong>TEST GIT CONNECTION</strong> and finally click <strong>SAVE &amp; CONTINUE</strong>.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36fcc100-cfe3-49b3-8ecd-8f85b7c4e1bd",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cba1ea5f-4021-4960-bac9-ba9bcc3c0faa",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>5. Create a Connection in ModelOps</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This step is optional if you plan to use the DEMO project which is created by default in ModelOps installations.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>There are 2 types of connections in ModelOps :</p>\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>User connections - These are used to validate model operationalization</li>\n",
    "    <li>Service connections - These are used to score models in Production</li>\n",
    "</ul>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>While creating the Project you will be asked to create your Service and Personal Connection.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can avoid creating a Service Connection for this Demo.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Click on <strong>NEXT</strong> button to skip to the \"Add Personal Connection\" sheet and click on the <strong>ADD PERSONAL CONNECTION</strong> button.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9166e2f-aff6-4bb1-b7ac-a6c92bb66cc1",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Include your Connection Details to Vantage</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "999e8c57-46b6-4ae5-8008-b5326db7f114",
   "metadata": {},
   "source": [
    "<img src=\"images/06_06.png\" alt=\"Add Personal Connection\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6524f5c5-a046-464b-94db-bbc4fe7e5746",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Include:</p>\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li><strong>Name:</strong> Demo User Personal</li>\n",
    "    <li><strong>Description:</strong> ClearScape Analytics Experience</li>\n",
    "    <li><strong>Host:</strong> ClearScape url</li>\n",
    "    <li><strong>Database:</strong> \"demo_user\"</li>\n",
    "    <li><strong>VAL Database:</strong> VAL</li>\n",
    "    <li><strong>BYOM Database:</strong> MLDB</li>\n",
    "    <li><strong>Login Mech:</strong> TDNEGO</li>\n",
    "    <li><strong>Username/Password</strong></li>\n",
    "</ul>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can test the connection by clicking the <strong>TEST CONNECTION</strong> button.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>And finally click the <strong>SAVE</strong> at the top.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8357b910-9f47-419c-8c5f-2b93cb0e9f56",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51b04210-1565-4d4d-96bb-6ba7456d7ad0",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>6. Run Health Checks on the connection (Optional)</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>This step is optional and not required if you plan to use the DEMO project which is created by default in ModelOps installations.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad1597f0-f5fd-4dac-9d38-d306336222c5",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Once you've created the new project and connection, you need to follow the steps:</p>\n",
    "\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li>Navigate to the newly created project, by clicking on the row of the project in the **Projects** screen.</li>\n",
    "    <li>Click on the <b>Settings</b> <img src=\"images/icons8-settings-24.png\" alt=\"Settings\" /> in the left sidebar at the bottom</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "341004fc-87ab-4b1f-996f-360e299f4636",
   "metadata": {},
   "source": [
    "<img src=\"images/06_07.png\" alt=\"Settings\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d504d9b-b318-42dc-87f7-03d644bca9d4",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Once in the \"Settings\" screen, click the <img src=\"images/icons8-menu-vertical-24.png\" alt=\"Menu\" /> button, right at the end of the connection.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then select \"View Details\" option from the dropdown menu.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90696b23-f1b9-41c7-9a96-c4d1facf4c70",
   "metadata": {},
   "source": [
    "<img src=\"images/06_08.png\" alt=\"View Details\" width=\"400\" height=\"400\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21bbf280-c50b-4123-b5f5-c95171d22b28",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Finally you should get the Health check details of the connection - checking permspace, VAL and BYOM access:</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2a13a7e-045a-4dd8-bba3-f1bbafa3a8eb",
   "metadata": {},
   "source": [
    "<img src=\"images/06_09.png\" alt=\"Health Check\" width=\"600\" height=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc024e3-7300-457a-a2fe-bf2a5eb9e922",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e81e871-91a9-4fcc-9b70-f122da63dc20",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>7. Create Dataset in ModelOPs</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>ModelOps requires to identify the Dataset in the UI application in order to operationalize the models. We will use the same queries used in this notebook to create the dataset instance for training and evaluation.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>First we need to create a Dataset Template which will enable us to create multiple instances for training/evaluation easily and faster.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69f7d1d-6784-421c-969f-dade77046ebc",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>7.1 Create Dataset Template</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go to Menu -> Datasets, and then click on Create Dataset template</p>\n",
    "\n",
    "<img src=\"images/05_01.png\" alt=\"Datasets\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Fill the fields for the Catalog. Use:</p>\n",
    "\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li><strong>Name:</strong> PIMA Dataset template</li>\n",
    "    <li><strong>Description:</strong> PIMA Dataset template</li>\n",
    "    <li><strong>Feature Catalog:</strong> Vantage</li>\n",
    "    <li><strong>Database:</strong> your-db</li>\n",
    "    <li><strong>Table:</strong> aoa_statistics_metadata</li>\n",
    "</ul>\n",
    "\n",
    "<img src=\"images/05_02.png\" alt=\"Catalog Dataset template\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Click next to provide the Features Query</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Features Query:</p>\n",
    "\n",
    "```sql\n",
    "SELECT * FROM pima_patient_features\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You may want to run it and preview data</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>By default the entity key is selected: PatientID and the rest of columns of the query as features. Keep it like that</p>\n",
    "\n",
    "<img src=\"images/05_03.png\" alt=\"Features Dataset template\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Continue to Entity & Target and include the query:</p>\n",
    "\n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses as F WHERE F.patientid MOD 5 <> 0\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You may want to run it and preview data</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select the Target variable: HasDiabetes</p>\n",
    "\n",
    "<img src=\"images/05_04.png\" alt=\"Features Dataset template\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Continue to Predictions and include the details of the database, table, and the query:</p>\n",
    "\n",
    "<ul style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li><strong>Database:</strong> your-db</li>\n",
    "    <li><strong>Table:</strong> pima_patient_predictions</li>\n",
    "    <li><strong>Query:</strong></li>\n",
    "</ul>\n",
    "\n",
    "```sql\n",
    "SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0\n",
    "```\n",
    "\n",
    "\n",
    "<img src=\"images/05_05.png\" alt=\"Features Dataset template\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdeddd38-0f1a-46af-8726-728315689c69",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now it's turn to create the instances for Training and Evaluation</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Get inside the dataset template and create a new instance.</p>\n",
    "\n",
    "<img src=\"images/05_06.png\" alt=\"Training Dataset instance\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select training and create it</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then include a name for the dataset instance and click next:</p>\n",
    "\n",
    "<img src=\"images/05_07.png\" alt=\"Training Dataset instance\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Confirm the query to get the dataset population for training:</p>\n",
    "\n",
    "<img src=\"images/05_08.png\" alt=\"Training Dataset instance\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now go back to the template and create a new instance.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Select evaluation and create it.</p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then include a name for the dataset instance and click next:</p>\n",
    "\n",
    "<img src=\"images/05_09.png\" alt=\"Evaluation Dataset instance\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Confirm the query to get the dataset population for evaluation:</p>\n",
    "\n",
    "<img src=\"images/05_10.png\" alt=\"Evaluation Dataset instance\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "473fa740-b6d5-45f7-a2ea-1b6cffe56401",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5c10073-a479-4c18-961c-e06feb21684b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<b style = 'font-size:20px;font-family:Arial;color:#00233C'>8. Cleanup</b>\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <p style = 'font-size:16px;font-family:Arial;color:#00233C'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf95b26c-02bc-495c-8759-ba41e25b5b92",
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c0bee48-3bb1-4d39-9cf4-4d643ef9bbdb",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62fc78de-172c-4426-81ac-011326623902",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b526c448-1ad5-4044-9273-96335bf020bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43cc7460-74fc-4211-bc01-04520cbf4e90",
   "metadata": {},
   "source": [
    "[<< Back to PIMA H2O BYOM](./05_ModelOps_BYOM_PIMA_H2O.ipynb) | [Continue to Git PIMA InDB GLM >>](./07_ModelOps_GIT_PIMA_Python_Indb_GLM.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "349eb35c-2d5a-4779-8816-0ec2eac23fc1",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}