UseCases/Data_Science_101_with_Python/Data_Science_101_with_Python_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       Data Science Flow 101\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>Introduction</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In this Jupyter Notebook, we will walk you through the standard flow of Data Science using the CRISP-DM standard (Cross-Industry Standard Process for Data Mining):</p>\n",
    "\n",
    "<img src=\"images/800px-CRISP-DM_Process_Diagram.png\"  alt=\"CRISP-DM\" style=\"width: 300px;\"/>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You can find more information on <a href=\"https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining\">Wikipedia</a> about this framework.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b> 1. Business Understanding </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The first step in CRISP-DM is the Business Understanding. Here, we want to better understand the impact of COVID-19 in Italy. To be precise, we would like to understand the maximum number of new daily cases that we will have in Italy.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>To forecast the number of cases, we will use the framework of Diffusion of Innovations where the number of people purchasing a new product (e.g. iPhone 11) or contracting a virus such as COVID-19 is proportional to the number of people who already have it (product or virus) and to the number of people left (word of mouth, community spreading) as well as extragenous factor which will be proportional to the number of people left.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>(New People) = (some parameter) * (Number of People who already have it) * (Number of People left who don't have it) + (some other parameter) * (Number of People left who don't have it) </p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>dN = a * N * (NMAX - N) + b * (NMAX - N)</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>with dN = New People (new cases), N = Total People with it (cumulative cases), NMAX = Total people in the market who could have it, and a & b  some parameters that we need to model.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Transforming this function and you get a basic 2nd order polynomial, i.e. f(X) = A + B * x + C * x^2 with x = N</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>2. Data Understanding</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>There are many sources of data. Here, we will use the data from the European Centre for Disease Prevention and Control at https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The current data in Excel is located at https://www.ecdc.europa.eu/sites/default/files/documents/COVID-19-geographic-disbtribution-worldwide-2020-03-10.xls. </p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We start by importing the required libraries and connecting to the database. You will be asked to enter the password.</p>    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%capture\n",
    "#!pip install --user iminuit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'><b>Note: </b><i>The above statements may need to be uncommented if you run the notebooks on a platform other than ClearScape Analytics Experience that does not have the libraries installed. If you uncomment those installs, be sure to restart the kernel after executing those lines to bring the installed libraries into memory. The simplest way to restart the Kernel is by typing zero zero: <b> 0 0</b></i></p>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import getpass\n",
    "from teradataml import *\n",
    "# Import Python wrapper on CERN's Minuit to fit a function using the Chi Square Minimalization approach\n",
    "from iminuit import Minuit, describe\n",
    "# Import NumPy to numeric computation\n",
    "import numpy as np\n",
    "# Import Matplotlib for charts\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "display.max_rows=5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=DataScienceFlow.ipynb;' UPDATE FOR SESSION;''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>3. Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We have provided data for this demo on cloud storage.  You have the option of either running the demo using foreign tables to access the data without using any storage on your environment or downloading the data to local storage which may yield somewhat faster execution, but there could be considerations of available storage.  There are two statements in the following cell, and one is commented out.  You may switch which mode you choose by changing the comment string. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call get_data('DEMO_DataScienceFlow_cloud');\"\n",
    " # Takes about 20 seconds\n",
    "# %run -i ../run_procedure.py \"call get_data('DEMO_DataScienceFlow_local');\" \n",
    "# Takes about 40 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Next is an optional step – if you want to see status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call space_report();\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Get the data from Vantage in the DataFrame.</p>    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qry = '''select * from DEMO_DataScienceFlow.Covid_Data;'''\n",
    "df = DataFrame.from_query(qry)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then we explore the data to better understand it. What is inside the dataframe?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>From this, we can guess/infer the following:</p>\n",
    "\n",
    "<ol style = 'font-size:16px;font-family:Arial;color:#00233C'>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>DateRep is the date of the reported values</li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>CountryExp is the country</li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>NewConfCases is the number of new cases for that day in that country</li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>NewDeaths is the number of new deaths for that day in that country</li>\n",
    "    <li style = 'font-size:16px;font-family:Arial;color:#00233C'>GeoId and EU are metadata information related to the country</li>\n",
    " </ol>   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>4. Data Preparation</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now that we understand what each columns and column's values mean, we can manipulate the dataframe to get the analytics dataset that we will use for the modelling</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's filter down to Italy. First, we create a list of boolean to check if the row is for Italy:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=df.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['CountryExp']=='Italy'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Then we apply this list to the current dataframe:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['CountryExp']=='Italy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We will now also remove all the zeros before the virus reached Italy with the same technique:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df['NewConfCases']>0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's order now by day:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=df.sort_values(by=['DateRep'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's add the cumulative number of cases and deaths with:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"CumulConfCases\"] = df[\"NewConfCases\"].cumsum()\n",
    "df[\"CumulDeaths\"] = df[\"NewDeaths\"].cumsum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Finally, we filter down the dataset to the columns of interest while renaming it dN and N:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases = df.filter(items=['DateRep', 'NewConfCases','CumulConfCases']) \\\n",
    "    .rename(columns={\"DateRep\": \"T\", \"NewConfCases\": \"dN\", \"CumulConfCases\": \"N\"})\n",
    "df_deaths = df.filter(items=['DateRep', 'NewDeaths','CumulDeaths']) \\\n",
    "    .rename(columns={\"DateRep\": \"T\", \"NewDeaths\": \"dN\", \"CumulDeaths\": \"N\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Finally, we define the time series index of those two final datasets:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases = df_cases.set_index('T')\n",
    "df_deaths = df_deaths.set_index('T')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's do some charts.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The time-series for N and dN:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases.plot(y=[\"N\",\"dN\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's plot the data in a scatter plot dN as function of N, as defined in our Diffusion of Innovation framework:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases.plot.scatter(x='N',y='dN')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>5. Modelling </b></p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>As seen in the scatter chart above, there is no sign of the data trending. This means that there is no sensitivity to how big NMAX will ever be (i.e. the total number of people that will be eventually infected).</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>To make a prediction, let's define this as 1% of Italy's population (1% of 60.5 million)</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NMAX = 60.5e6 * 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(a, b):\n",
    "    return a * (NMAX - df_cases['N']) + b * df_cases['N'] * (NMAX - df_cases['N'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We define the error function (i.e. Chi Square) which return the sum of the error squared for each points:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chisquare(a, b):    \n",
    "    return (df_cases['dN'] - fit(a,b)).pow(2).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>For example, the total Chi Square error for a = 0.1 and b = 0.2 is </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chisquare(0.1, 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We now create the minimization object m with Minuit to minimize the function 'chisquare' </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = Minuit(chisquare, 0.1, 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We now execute the fit to find the best parameters describing the data:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.migrad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>6. Evaluation</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Now that we have our model with parameters, we can check the robustness of our model. We can do this multiple way.</p> \n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>In cases, we have a lot of data, we can use a test sample, or cross-validate with a different time range or market.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, with very limited number, we can do basic assessment of the error of our parameters.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We can start looking at this Chi Square function as function of the parameter `a` and identify the range in which we can trust its value (i.e. confidence interval).</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.draw_mnprofile(\"a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Ballpark, the best value from the data is 5.85e-5, but there is 68% chance that the value is between 5.7e-5 and 5.9e-5.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We can now check the parameter `b`:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.draw_mnprofile(\"b\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here the best value is 3.231E-7, but there is 68% chance that the value is between 3.22e-7 and 3.233e-7.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>We can check the dependency between the two variables:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.draw_mncontour(\"a\",\"b\", cl = [0.680, 0.950])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Here, the black ellipse represents the range we just identified at 68% confidence interval. The red ellipse is at 95% confidence interval.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's now apply those fitted value to see the result in a chart:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a=m.values[\"a\"]\n",
    "b=m.values[\"b\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases['fit'] = a * (NMAX - df_cases['N']) + b * df_cases['N'] * (NMAX - df_cases['N'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cases.plot('N',['dN','fit'],style=['o','-'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>7. Deployment</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Deployment is about using the insights we found to do something differently. It can be a single event or a repetitive tasks.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Let's start with the simpler case. For one off case such as here, we would just extrapolate the insights into the answer we are looking for: what is the maximum daily new cases we can have in Italy.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xx=np.arange(0,NMAX*0.55,NMAX/1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yy = a * (NMAX - xx) + b * xx * (NMAX - xx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(df_cases['N'], df_cases['dN'], '^', xx, yy, '-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Assuming all our assumptions are correct (which they are likely not), this means that at the peak of the epidemic in Italy, we could see 30,000 new cases per day.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>For repetitive tasks, we will need to automate the analysis and the deployment of the insights. \n",
    "So, we can start by developing one Python script with only the necessary steps, as well as input parameter to collect \n",
    "the relevant date of report, country, assumption, etc.</p>\n",
    "\n",
    "<p style = 'font-size:14px;font-family:Arial;color:#00233C'>Here is an example of a covid19.py script:</p>\n",
    "\n",
    "```python\n",
    "#!//usr/bin/python3\n",
    "import sys\n",
    "import pandas as pd\n",
    "from iminuit import Minuit, describe\n",
    "import numpy as np\n",
    "\n",
    "# Get Report Date from command line\n",
    "if len(sys.argv) != 3:\n",
    "        print('Usage: covid19.py [date] [country]')\n",
    "        sys.exit(1)\n",
    "rep_date = sys.argv[1]\n",
    "country = sys.argv[2]\n",
    "\n",
    "# Import data locally\n",
    "data_url = \"https://www.ecdc.europa.eu/sites/default/files/documents/COVID-19-geographic-disbtribution-worldwide-\" + rep_date + \".xls\"\n",
    "df = pd.read_excel(data_url)\n",
    "\n",
    "# Filter data\n",
    "df = df[df['CountryExp']==country]\n",
    "df = df[df['NewConfCases']>0]\n",
    "df = df.sort_values(by=['DateRep'])\n",
    "df[\"CumulConfCases\"] = df[\"NewConfCases\"].cumsum()\n",
    "df[\"CumulDeaths\"] = df[\"NewDeaths\"].cumsum()\n",
    "df = df.filter(items=['DateRep', 'NewConfCases','CumulConfCases']). \\\n",
    "    rename(columns={\"DateRep\": \"T\", \"NewConfCases\": \"dN\", \"CumulConfCases\": \"N\"}). \\\n",
    "    set_index('T')\n",
    "\n",
    "# Prepare model fit\n",
    "NMAX = 60.5e6 * 0.01\n",
    "x = df['N']\n",
    "y = df['dN']\n",
    "\n",
    "def fit(a, b):\n",
    "    return a * (NMAX - x) + b * x * (NMAX - x)\n",
    "def chisquare(a, b):\n",
    "    return (y - fit(a,b)).pow(2).sum()\n",
    "\n",
    "# Execute the fit\n",
    "m = Minuit(chisquare, print_level=0, pedantic=False)\n",
    "m.migrad()\n",
    "a=m.values[\"a\"]\n",
    "b=m.values[\"b\"]\n",
    "xx=np.arange(0,NMAX,NMAX/100)\n",
    "yy = a * (NMAX - xx) + b * xx * (NMAX - xx)\n",
    "print(\"Max:\",max(yy))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Running ```./covid19.py 2020-03-10 Italy``` will give ```Max: 29582.958532824043```.</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>Whereas ```./covid19.py 2020-03-08 Italy``` will give ```Max: 29105.409956316485```.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>With this script doing the work, we would need to deploy it on a production system, for example a Docker image, AWS EC2, or\n",
    "other Google GCE and now may be ModelOps in Teradata Vantage. Then set up a scheduler to refresh the analysis, a monitoring to alert when there is an error, and finally \n",
    "integrate in the operation. What will you do with this data? Maybe here an automated email to the secretary of health in Italy with the forecast of the worst to come?</p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>And you, what do you think? What would you do?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;background-color:#00233C;\">\n",
    "\n",
    "<p style = 'font-size:20px;font-family:Arial;color:#00233C'><b>8. CleanUp</b></p>\n",
    "<p style = 'font-size:18px;font-family:Arial;color:#00233C'><b>Databases and Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial;color:#00233C'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../run_procedure.py \"call remove_data('DEMO_DataScienceFlow');\" \n",
    "#Takes 5 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<footer style=\"padding-bottom:35px; background:#f9f9f9; border-bottom:3px solid #00233C\">\n",
    "    <div style=\"float:left;margin-top:14px\">ClearScape Analytics™</div>\n",
    "    <div style=\"float:right;\">\n",
    "        <div style=\"float:left; margin-top:14px\">\n",
    "            Copyright © Teradata Corporation - 2023. All Rights Reserved\n",
    "        </div>\n",
    "    </div>\n",
    "</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "toc-autonumbering": true,
  "toc-showtags": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}