image_classifier_pytorch_challenge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing an AI application\n",
    "\n",
    "Going forward, AI algorithms will be incorporated into more and more everyday applications. For example, you might want to include an image classifier in a smart phone app. To do this, you'd use a deep learning model trained on hundreds of thousands of images as part of the overall application architecture. A large part of software development in the future will be using these types of models as common parts of applications. \n",
    "\n",
    "In this project, you'll train an image classifier to recognize different species of flowers. You can imagine using something like this in a phone app that tells you the name of the flower your camera is looking at. In practice you'd train this classifier, then export it for use in your application. We'll be using [this dataset](http://www.robots.ox.ac.uk/~vgg/data/flowers/102/index.html) of 102 flower categories, you can see a few examples below. \n",
    "\n",
    "<img src='assets/Flowers.png' width=500px>\n",
    "\n",
    "The project is broken down into multiple steps:\n",
    "\n",
    "* Load and preprocess the image dataset\n",
    "* Train the image classifier on your dataset\n",
    "* Use the trained classifier to predict image content\n",
    "\n",
    "We'll lead you through each part which you'll implement in Python.\n",
    "\n",
    "When you've completed this project, you'll have an application that can be trained on any set of labeled images. Here your network will be learning about flowers and end up as a command line application. But, what you do with your new skills depends on your imagination and effort in building a dataset. For example, imagine an app where you take a picture of a car, it tells you what the make and model is, then looks up information about it. Go build your own dataset and make something new.\n",
    "\n",
    "First up is importing the packages you'll need. It's good practice to keep all the imports at the beginning of your code. As you work through this notebook and find you need to import a package, make sure to add the import up here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# %%capture suppresses jupyter output for this cell\n",
    "# make sure required packages are installed -- before imports\n",
    "# udacity installed version is 0.4.1; so this forces upgrade to eliminate env. issues\n",
    "!pip install torch==1.0.0  \n",
    "!pip install torchsummary\n",
    "!pip install seaborn\n",
    "!pip install pillow\n",
    "!pip install wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports here\n",
    "import torch                    # version 0.4.1 for Udacity\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import torch.optim as optim\n",
    "from torch.optim.lr_scheduler import MultiStepLR\n",
    "\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.sampler import RandomSampler \n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "\n",
    "import torchvision                       # ACTION - do we need all items?\n",
    "import torchvision.models as models\n",
    "import torchvision.transforms as transforms\n",
    "from torchvision import datasets         # ACTION - is this needed?  I think not since we are importing folder of images\n",
    "from torchvision.datasets import ImageFolder\n",
    "\n",
    "from torchsummary import summary\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# visualization imports\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.mlab as mlab\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set(style=\"darkgrid\")\n",
    "\n",
    "# various utilities\n",
    "import json\n",
    "import os\n",
    "from collections import Counter  # for counting images and histograms\n",
    "import random\n",
    "import gc                        # python memory garbage collection, to stay clean\n",
    "import wget                      # retrieving source data\n",
    "import zipfile\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=UserWarning)\n",
    "\n",
    "# for unpacking `state_dict`\n",
    "from collections import OrderedDict \n",
    "\n",
    "# PIL image processing\n",
    "from PIL import Image\n",
    "\n",
    "# check if CUDA is available\n",
    "train_on_gpu = torch.cuda.is_available()\n",
    "\n",
    "# global setting to use gpu or cpu based on available hardware\n",
    "if not train_on_gpu:\n",
    "    device = torch.device('cpu')\n",
    "    print('\\nCUDA is NOT available.  Training on CPU ...')\n",
    "else:\n",
    "    device = torch.device('cuda')\n",
    "    print('CUDA is available!  Training on GPU ...')\n",
    "    \n",
    "# setup slack notifications; required SLACK vars in your env variables\n",
    "import os\n",
    "from slackclient import SlackClient\n",
    "token = os.environ['SLACK_API_TOKEN']\n",
    "channel_id = os.environ['SLACK_CHANNEL_ID']\n",
    "\n",
    "import datetime\n",
    "import json\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# confirm token exists in current system\n",
    "'SLACK_API_TOKEN' in os.environ\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# confirm channel ID exists in current system \n",
    "'SLACK_CHANNEL_ID' in os.environ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SlackClient(token)\n",
    "\n",
    "def slack_message(message, attachments, channel_id):\n",
    "    sc = SlackClient(token)\n",
    "    sc.api_call('chat.postMessage', \n",
    "                channel=channel_id, \n",
    "                text=message,\n",
    "                attachments = attachments,\n",
    "                username='surfaceowl',\n",
    "                icon_emoji=':robot_face:')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "date = datetime.datetime.now().strftime(\"%x\")\n",
    "day_of_week = datetime.datetime.now().strftime(\"%A\")\n",
    "\n",
    "# custom_message = f\"manual test of slackclient in jupyter notebook sent on { day_of_week }, { date}\"\n",
    "# slack_message(custom_message, [], channel_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Global variables - required by Udacity\n",
    "\n",
    "# ACTION - LIST ASSUMPTIONS IN MARKDOWN CELL, EARLY\n",
    "num_classes = 102          # we calc this locally, but source data not provided in udacity\n",
    "norm_mean = [0.485, 0.456, 0.406]\n",
    "norm_std = [0.229, 0.224, 0.225]\n",
    "\n",
    "# global variables\n",
    "global_saved_file_name = 'model_flowers.v60.pt'\n",
    "\n",
    "# create manual flag to force to run on cpu\n",
    "# helps us pretest inference on local machine cpu which may have a gpu\n",
    "# because Udacity environment is cpu only (avoids surprises on submit)\n",
    "force_train_on_cpu = False\n",
    "\n",
    "# number of subprocesses for data loading; use almost all available cpu cores\n",
    "num_workers = os.cpu_count() -1\n",
    "\n",
    "\n",
    "# machine learning variables\n",
    "batch_size = 64\n",
    "\n",
    "# scheduler - parameters to change learning rate over epochs\n",
    "milestones = [20, 50, 75]\n",
    "gamma = 0.1\n",
    "\n",
    "# future user inputs\n",
    "learning_rate = 0.001\n",
    "\n",
    "hidden_layers = \"Manual\" # choose [Zero, Manual, Auto]\n",
    "# experiment on parameterizing number of hidden layers\n",
    "hidden_layers1 = [2208, 512]\n",
    "hidden_layers2 = [512, 256]\n",
    "hidden_layer_sizes = list(zip(hidden_layers1, hidden_layers2))\n",
    "print(list(hidden_layer_sizes))\n",
    "\n",
    "\n",
    "n_epochs = 30\n",
    "\n",
    "# ACTION - clean up available models; requires manual investigation of last layer\n",
    "available_models = {\"vgg16\":25088,\n",
    "                    \"densenet121\" : 1024,\n",
    "                    \"densenet161\": 2208,\n",
    "                    \"resnet152\" : 2048 }\n",
    "\n",
    "chosen_model = \"densenet161\"\n",
    "\n",
    "\n",
    "# image processing\n",
    "image_size_required = 224\n",
    "image_size = 224\n",
    "mean_train_imgs = [0.485, 0.456, 0.406]\n",
    "std_train_imgs = [0.229, 0.224, 0.225]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "\n",
    "Here you'll use `torchvision` to load the data ([documentation](http://pytorch.org/docs/0.3.0/torchvision/index.html)). You can [download the data here](https://s3.amazonaws.com/content.udacity-data.com/courses/nd188/flower_data.zip). The dataset is split into two parts, training and validation. For the training, you'll want to apply transformations such as random scaling, cropping, and flipping. This will help the network generalize leading to better performance. If you use a pre-trained network, you'll also need to make sure the input data is resized to 224x224 pixels as required by the networks.\n",
    "\n",
    "The validation set is used to measure the model's performance on data it hasn't seen yet. For this you don't want any scaling or rotation transformations, but you'll need to resize then crop the images to the appropriate size.\n",
    "\n",
    "The pre-trained networks available from `torchvision` were trained on the ImageNet dataset where each color channel was normalized separately. For both sets you'll need to normalize the means and standard deviations of the images to what the network expects. For the means, it's `[0.485, 0.456, 0.406]` and for the standard deviations `[0.229, 0.224, 0.225]`, calculated from the ImageNet images.  These values will shift each color channel to be centered at 0 and range from -1 to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO: load data, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define source data directories, use os.path.join to avoid environment path issues\n",
    "data_dir = os.path.join(os.getcwd(), \"flower_data\")\n",
    "train_dir = os.path.join(data_dir, \"train\")\n",
    "valid_dir = os.path.join(data_dir, \"valid\")\n",
    "test_dir = os.path.join(data_dir, \"test\")\n",
    "\n",
    "# check if source data exists\n",
    "data_dir_exists = os.path.isdir(data_dir)  # returns True/False\n",
    "\n",
    "# if source data does not exist, download source file - then manually unzip \n",
    "if not data_dir_exists:\n",
    "    sourcedata_url = 'https://s3.amazonaws.com/content.udacity-data.com/courses/nd188/flower_data.zip'\n",
    "    filename = wget.download(sourcedata_url)\n",
    "\n",
    "# check if test data provided; make a variable for conditional checks below\n",
    "test_data_exists = os.path.isdir(test_dir)\n",
    "print(\"test_data_exists: \", test_data_exists)  # returns True/False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.path.join(data_dir, \"train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label mapping\n",
    "\n",
    "You'll also need to load in a mapping from category label to category name. You can find this in the file `cat_to_name.json`. It's a JSON object which you can read in with the [`json` module](https://docs.python.org/2/library/json.html). This will give you a dictionary mapping the integer encoded categories to the actual names of the flowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('cat_to_name.json', 'r') as f:\n",
    "    cat_to_name = json.load(f)\n",
    "\n",
    "# print initial cat_to_name; then sort - easier to understand  categories\n",
    "print(list(cat_to_name.items())[:10])\n",
    "cat_to_name = list((int(key), value) for (key,value) in cat_to_name.items())\n",
    "\n",
    "num_classes = len(cat_to_name)\n",
    "print(\"number of classes = \", num_classes)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# create dataframe mapping directories->classes & counting images\n",
    "class_matrix = pd.DataFrame(columns={\"dir_num\", \"flower_name\", \"num_images\"})\n",
    "\n",
    "map_sorted = []\n",
    "image_stats = []\n",
    "\n",
    "for index, category in enumerate(cat_to_name):\n",
    "    # summarize # of images by directory/flower\n",
    "    dir_num = int(cat_to_name[index][0])\n",
    "    flower_name = cat_to_name[index][1]\n",
    "    target_directory = os.path.join(train_dir, str(dir_num))\n",
    "    num_images = int(len(os.listdir(target_directory)))\n",
    "    row = [dir_num, flower_name, num_images]\n",
    "    map_sorted.append(row)\n",
    "    \n",
    "    # get image sizes for every image\n",
    "    for image in os.listdir(target_directory):\n",
    "        target_image = os.path.join(target_directory, image)\n",
    "        image = Image.open(target_image)\n",
    "        image_array = np.array(image)\n",
    "        # get size of each image\n",
    "        image_height = image_array.shape[0]\n",
    "        image_width = image_array.shape[1]\n",
    "        image_stats.append([dir_num, flower_name, image, \n",
    "                            image_width, image_height])\n",
    "\n",
    "# create some handy dataframes\n",
    "class_matrix = pd.DataFrame(data=map_sorted)\n",
    "class_matrix = class_matrix.rename(mapper={0: \"dir_num\", \n",
    "                                           1: \"flower_name\", \n",
    "                                           2: \"num_images\"}, axis=1)\n",
    "\n",
    "class_matrix[\"mean_num_images\"] = class_matrix[\"num_images\"].mean()\n",
    "class_matrix[\"mean_num_images\"] = class_matrix[\"mean_num_images\"].astype(int)\n",
    "class_matrix.head(5)\n",
    "\n",
    "df_image_details = pd.DataFrame(data=image_stats)\n",
    "df_image_details = df_image_details.rename(mapper={0: \"dir_num\", \n",
    "                                               1: \"flower_name\", \n",
    "                                               2: \"image_file\", \n",
    "                                               3: \"width\", \n",
    "                                               4: \"height\"}, axis=1)\n",
    "\n",
    "# TODO: extract number of images by class into list; for use in DataLoader weighting\n",
    "# class_sample_counts = class_matrix[\"num_images\"].values.tolist()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see distribution of training data\n",
    "labels = class_matrix[\"flower_name\"]\n",
    "values = class_matrix[\"num_images\"]\n",
    "means = class_matrix[\"mean_num_images\"]\n",
    "\n",
    "plt.figure(figsize=(18,4))\n",
    "plt.title(\"# of images per flower\")\n",
    "ax = sns.barplot(labels, values, alpha = 0.9)\n",
    "plt.bar(labels, values)\n",
    "sns.lineplot(labels, means, color='#bc5145', \n",
    "             label=f'mean = {class_matrix[\"mean_num_images\"].max()}')\n",
    "plt.xticks(labels, rotation=90)\n",
    "ax.set(xlabel='flower', ylabel='# of images')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_image_stats = df_image_details.loc[:, [\"flower_name\", \"width\", \"height\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df_image_stats = df_image_stats.groupby(\"flower_name\").describe()\n",
    "df_image_stats.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# finding:\n",
    "# all images are at least 500px X 500px"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define transformations and datasets\n",
    "# important: parameterize final image size; pretrained models use different sizes\n",
    "\n",
    "# Training data - gets augmentation\n",
    "\n",
    "data_transforms_train = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.RandomRotation(35),\n",
    "    transforms.CenterCrop(size=image_size_required),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean_train_imgs, std_train_imgs)\n",
    "    ])\n",
    "\n",
    "# Validation data - no augmentation\n",
    "data_transforms_valid = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(size=image_size_required),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean_train_imgs, std_train_imgs)\n",
    "    ])\n",
    "\n",
    "# Test data - no augmentation\n",
    "data_transforms_test = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(size=image_size_required),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean_train_imgs, std_train_imgs)\n",
    "    ])\n",
    "\n",
    "\n",
    "# Load the datasets with ImageFolder\n",
    "train_datasets = torchvision.datasets.ImageFolder(root=train_dir, transform=data_transforms_train)\n",
    "valid_datasets = torchvision.datasets.ImageFolder(root=valid_dir, transform=data_transforms_valid)\n",
    "\n",
    "\n",
    "# define dataloaders; note: shuffle=True, can't be used with sampling\n",
    "train_loader = torch.utils.data.DataLoader(train_datasets, \n",
    "                                           batch_size=batch_size, \n",
    "                                           num_workers=num_workers, \n",
    "                                           shuffle=True)\n",
    "                                           # rebalancing train happens here\n",
    "\n",
    "valid_loader = torch.utils.data.DataLoader(valid_datasets, \n",
    "                                           batch_size=batch_size,  \n",
    "                                           num_workers=num_workers)\n",
    "\n",
    "# workaround- no test data provided in challenge, but must check test for scoring\n",
    "# we'll create a conditional test and create loaders only if `test` data exists\n",
    "if test_data_exists:\n",
    "    test_datasets = torchvision.datasets.ImageFolder(root=test_dir, \n",
    "                                                     transform=data_transforms_test)\n",
    "    \n",
    "    test_sampler = RandomSampler(test_datasets, replacement=False)\n",
    "    \n",
    "    test_loader = torch.utils.data.DataLoader(test_datasets, \n",
    "                                              batch_size=batch_size, \n",
    "                                              sampler=test_sampler, \n",
    "                                              num_workers=num_workers)\n",
    "\n",
    "# summarize input we have to work with\n",
    "print(\"# of training images= \", len(train_datasets))\n",
    "print(\"# of validation images = \", len(valid_datasets))\n",
    "print(\"training data # of batches = \", len(train_loader))\n",
    "print(\"validation data # of batches = \", len(valid_loader))\n",
    "\n",
    "if test_data_exists:\n",
    "    print(\"# of test images = \", len(test_datasets))\n",
    "    print(\"test data # of batches = \", len(test_loader))\n",
    "else:\n",
    "    print(\"test data - none available - skipping\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create helper function to un-normalize and display an image\n",
    "\n",
    "def imageshow_numpy(image_numpy):\n",
    "    \"\"\"\n",
    "    Convert numpy image array to matplotlib format (fix color channel order), \n",
    "    normalize image with std & mean from training data & display image\n",
    "    Function named to distinguish from matplotlib imshow\n",
    "    \n",
    "    :param image: numpy array representing image\n",
    "    :return converted image\n",
    "    \"\"\"\n",
    "    # PyTorch assumes color channel first dimension\n",
    "    # but matplotlib assumes color channel third dimension; need to swap\n",
    "    image = np.transpose(image_numpy, (1, 2, 0))\n",
    "    \n",
    "    # normalize images based on training data std and mean (given above)\n",
    "    image = std_train_imgs * image + mean_train_imgs\n",
    "    \n",
    "    # clip to range to suppress warnings\n",
    "    image = np.clip(image, 0, 1)\n",
    "    \n",
    "    plt.imshow(image)\n",
    "    \n",
    "    return image\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print sample of a batch of training images\n",
    "image_iterator = iter(train_loader)\n",
    "images, labels = image_iterator.next()\n",
    "images = images.numpy() # convert images to numpy for display\n",
    "\n",
    "# plot the images with labels in the batch\n",
    "fig = plt.figure(figsize=(25, 16))\n",
    "\n",
    "for image_id in np.arange(1, 21, 1):\n",
    "    ax = fig.add_subplot(4, 20/4, image_id, xticks=[], yticks=[])\n",
    "    imageshow_numpy(images[image_id])\n",
    "    ax.set_title(cat_to_name[labels[image_id]])  # label with cat # and name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Humans - STOP Here\n",
    "*look at your data_transforms; \n",
    "\n",
    "**intuitively, could you identify these flowers as displayed above based on zoom/crop + other transforms?**\n",
    "\n",
    "*if not - adjust your data transformations*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building and training the classifier\n",
    "\n",
    "Now that the data is ready, it's time to build and train the classifier. As usual, you should use one of the pretrained models from `torchvision.models` to get the image features. Build and train a new feed-forward classifier using those features.\n",
    "\n",
    "We're going to leave this part up to you. If you want to talk through it with someone, chat with your fellow students! You can also ask questions on the forums or join the instructors in office hours.\n",
    "\n",
    "Refer to [the rubric](https://review.udacity.com/#!/rubrics/1663/view) for guidance on successfully completing this section. Things you'll need to do:\n",
    "\n",
    "* Load a [pre-trained network](http://pytorch.org/docs/master/torchvision/models.html) (If you need a starting point, the VGG networks work great and are straightforward to use)\n",
    "* Define a new, untrained feed-forward network as a classifier, using ReLU activations and dropout\n",
    "* Train the classifier layers using backpropagation using the pre-trained network to get the features\n",
    "* Track the loss and accuracy on the validation set to determine the best hyperparameters\n",
    "\n",
    "We've left a cell open for you below, but use as many as you need. Our advice is to break the problem up into smaller parts you can run separately. Check that each part is doing what you expect, then move on to the next. You'll likely find that as you work through each part, you'll need to go back and modify your previous code. This is totally normal!\n",
    "\n",
    "When training make sure you're updating only the weights of the feed-forward network. You should be able to get the validation accuracy above 70% if you build everything right. Make sure to try different hyperparameters (learning rate, units in the classifier, epochs, etc) to find the best model. Save those hyperparameters to use as default values in the next part of the project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### import pretrained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_pretrained_model(model_name):\n",
    "    \"\"\"\n",
    "    Downloads pretrained model from torchvision\n",
    "    :param model_name: name of pretrained model\n",
    "    :type model_name: str\n",
    "    \n",
    "    :return pytorch model using chosen architecture, ready to use or modify\n",
    "    \"\"\"\n",
    "    if model_name == 'vgg16':\n",
    "        model = models.vgg16(pretrained=True)\n",
    "    elif model_name == 'densenet121':\n",
    "        model = models.densenet121(pretrained=True)\n",
    "    elif model_name == 'densenet161':\n",
    "        model = models.densenet161(pretrained=True)\n",
    "    elif model_name == 'resnet152':\n",
    "        model = models.resnet152(pretrained=True)\n",
    "    else:\n",
    "        print(f\"{ model_name } is NOT implemented in this notebook.\"\n",
    "              \"The following models are available: vgg16, densenet121,or resnet152\")\n",
    "    \n",
    "    # append more explicit model_name for later use\n",
    "    model.model_name = model_name\n",
    "    \n",
    "    return model\n",
    "\n",
    "pretrained_model = download_pretrained_model(chosen_model)\n",
    "\n",
    "print(pretrained_model, \"\\n\\n\", pretrained_model.model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### define custom classifier "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a new, untrained feed-forward network as a classifier, using ReLU activations and dropout\n",
    "\n",
    "# ACTION - UPDATE\n",
    "# class Sequential(nn.Module):\n",
    "#    def __init__(self, n_inputs, n_classes):\n",
    "#        super(Sequential, self).__init__()\n",
    "#        self.fc1 = nn.Linear(n_inputs, int(hidden_layers))\n",
    "#        self.dropout_20 = nn.Dropout(p=.2)\n",
    "#        self.fc2 = nn.Linear(int(hidden_layers), n_classes)\n",
    "        \n",
    "#        # Zero Dropout layer      \n",
    "#        self.dropout_0 = nn.Dropout(p=0)\n",
    "        \n",
    "#        # Output layer - to give probabilities per class\n",
    "#        self.out = nn.Softmax(dim = 1)\n",
    "#        # ACTION:  self.out = nn.LogSoftmax(dim = 1)\n",
    "    \n",
    "#    def forward(self, x):\n",
    "        # add sequence of convolutional and max pooling layers -- ACTION - no convolutional or pooling is here.  FIX!\n",
    "#        x = self.fc1(x)\n",
    "#        x = self.dropout_20(F.relu(x))\n",
    "#        x = self.fc2(x)\n",
    "#        x=self.out(x)\n",
    "#        return x\n",
    "\n",
    "def create_custom_classifier(num_in_features, \n",
    "                             hidden_layers, \n",
    "                             num_out_features=num_classes):\n",
    "    \"\"\"\n",
    "    Function that defines our custom classifier\n",
    "    :param num_in_features: # of elements in next-to-last layer of pretrained network\n",
    "    :param hidden_layers: \n",
    "    :type hidden_layers: boolean\n",
    "    :param num_out_features: number of classifications to make; should equal number of possible classes\n",
    "\n",
    "    :return customized nn.Sequential classifier\n",
    "    \"\"\"\n",
    "    custom_classifier = nn.Sequential()\n",
    "    \n",
    "    if hidden_layers == \"Zero\":\n",
    "        custom_classifier.add_module('fc0', nn.Linear(num_in_features, 102))\n",
    "\n",
    "    elif hidden_layers == \"Manual\":\n",
    "            # add layer, matching # of input features to # of the first hidden layer\n",
    "        custom_classifier.add_module('fc0', nn.Linear(num_in_features, 512))\n",
    "        custom_classifier.add_module('relu0', nn.ReLU())\n",
    "        custom_classifier.add_module('drop0', nn.Dropout(.6))\n",
    "        custom_classifier.add_module('fc1', nn.Linear(512, 256))\n",
    "        custom_classifier.add_module('relu1', nn.ReLU())\n",
    "        custom_classifier.add_module('drop1', nn.Dropout(.5))\n",
    "        custom_classifier.add_module('output', nn.Linear(256, num_out_features))\n",
    "        \n",
    "    else:\n",
    "        # add layer, matching # of input features to # of the first hidden layer\n",
    "        custom_classifier.add_module('fc0', nn.Linear(num_in_features, hidden_layer_sizes[0][0]))\n",
    "        custom_classifier.add_module('relu0', nn.ReLU())\n",
    "        custom_classifier.add_module('drop0', nn.Dropout(.6))\n",
    "        custom_classifier.add_module('relu1', nn.ReLU())\n",
    "        custom_classifier.add_module('drop1', nn.Dropout(.5))\n",
    "        for i, layer_group in enumerate(hidden_layer_sizes):\n",
    "            custom_classifier.add_module('fc'+str(i+1), \n",
    "                                         nn.Linear(layer_group[0], \n",
    "                                                   layer_group[1]))\n",
    "            custom_classifier.add_module('relu'+str(i+1), nn.ReLU())\n",
    "            custom_classifier.add_module('drop'+str(i+1), nn.Dropout(.5))\n",
    "        custom_classifier.add_module('output', nn.Linear(layer_group[1], num_out_features))\n",
    "\n",
    "    return custom_classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# TODO: Build and train your network\n",
    "\n",
    "def freeze_layers(model):\n",
    "    \"\"\"\n",
    "    Function freezes layers so weights and biases are not updated in a model\n",
    "    required frequently, so make a function\n",
    "    :param model: a pytorch model we want to freeze\n",
    "    \n",
    "    : return frozen model\n",
    "    \"\"\"\n",
    "    for param in model.parameters():\n",
    "        param.requires_grad = False\n",
    "    \n",
    "    return model\n",
    "\n",
    "\n",
    "def setup_customize_model(pretrained_model, num_classes):\n",
    "    \"\"\"\n",
    "    Freezes layers and replaces classifiers for predefined torchvision models\n",
    "    examines each model, gets # of input/output features in last layer\n",
    "    so we can match those numbers up with the sizes in our custom classifier\n",
    "    :param model: a pretrained model that has already been downloaded\n",
    "    :param num_classes: the number of output classes for classification\n",
    "    :return model, optimizer, criterion - three neural net objects needed for training/evaluation\n",
    "    \"\"\"\n",
    "    \n",
    "    # ACTION - CONDENSE THIS IF SWITCH TO IN [LIST OF CASES]\n",
    "    model = pretrained_model\n",
    "    \n",
    "    # important notes\n",
    "    # 1- The last layer does  classification, but the NAME of the last layer differs between torchvision models\n",
    "    # so we need an IF switch to properly replace the last layer\n",
    "    # 2- get in_features of last layer as a parameter; and feed that to our custom classifier\n",
    "    # 3- order of operations:  important to freeze the model before replacing classifier\n",
    "    \n",
    "    if model.model_name == 'vgg16':\n",
    "        model.classifier_layer_name = 'classifier'\n",
    "        n_inputs = model.classifier[0].in_features\n",
    "        n_outputs = model.classifier[0].out_features\n",
    "        model = freeze_layers(model)  # freeze layers\n",
    "        \n",
    "        # replace classifier\n",
    "        model.classifier = create_custom_classifier(n_inputs,\n",
    "                                                    hidden_layers,\n",
    "                                                    num_classes)\n",
    "        criterion = nn.NLLLoss()\n",
    "        optimizer_target = model.classifier.parameters()\n",
    "\n",
    "    elif model.model_name == 'densenet121':\n",
    "        model.classifier_layer_name = 'classifier'\n",
    "        n_inputs = model.classifier.in_features\n",
    "        n_outputs = model.classifier.out_features\n",
    "        model = freeze_layers(model)  # freeze layers\n",
    "        \n",
    "        # replace classifier\n",
    "        model.classifier = create_custom_classifier(n_inputs,\n",
    "                                                    hidden_layers,\n",
    "                                                    num_classes)\n",
    "        criterion = nn.CrossEntropyLoss()\n",
    "        optimizer_target = model.classifier.parameters()\n",
    "        \n",
    "    elif model.model_name == 'densenet161':\n",
    "        model.classifier_layer_name = 'classifier'\n",
    "        n_inputs = model.classifier.in_features\n",
    "        n_outputs = model.classifier.out_features\n",
    "        model = freeze_layers(model)  # freeze layers\n",
    "        \n",
    "        # replace classifier\n",
    "        model.classifier = create_custom_classifier(n_inputs,\n",
    "                                                    hidden_layers,\n",
    "                                                    num_classes)\n",
    "        criterion = nn.CrossEntropyLoss()\n",
    "        optimizer_target = model.classifier.parameters()\n",
    "\n",
    "    elif model.model_name == 'resnet152':\n",
    "        model.classifier_layer_name = 'fc'\n",
    "        n_inputs = model.fc.in_features\n",
    "        n_outputs = model.fc.out_features\n",
    "        model = freeze_layers(model)  # freeze layers\n",
    "        \n",
    "        # replace classifier; note different name\n",
    "        model.fc = create_custom_classifier(n_inputs,\n",
    "                                            hidden_layers,\n",
    "                                            num_classes)\n",
    "        criterion = nn.CrossEntropyLoss()\n",
    "        optimizer_target = model.fc.parameters()\n",
    "\n",
    "    # setup criterion and optimizer\n",
    "    # notes: CrossEntropyLoss can be helpful when unbalanced training set\n",
    "    # https://pytorch.org/docs/stable/nn.html\n",
    "        \n",
    "    # setup optimizer\n",
    "    # option below required for udacity\n",
    "    # optimizer = optim.Adam(filter(lambda p: p.requires_grad, \n",
    "    #                       model.parameters()), \n",
    "    #                       lr=learning_rate)\n",
    "\n",
    "    optimizer = optim.Adadelta(model.parameters())\n",
    "    \n",
    "    return model, optimizer, criterion \n",
    "    \n",
    "\n",
    "model, optimizer, criterion = setup_customize_model(pretrained_model,\n",
    "                                                   num_classes)\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "print(f\"{total_params:,} - total possible parameters.\")\n",
    "total_trainable_params = sum(\n",
    "    p.numel() for p in model.parameters() if p.requires_grad)\n",
    "print(f\"{total_trainable_params:,}     training parameters will be used.\")\n",
    "\n",
    "model.total_trainable_params = total_trainable_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### map classes to indices and vice versa, so we can save as part of model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.class_to_idx = train_datasets.class_to_idx\n",
    "model.idx_to_class = {\n",
    "    idx: class_\n",
    "    for idx, class_ in cat_to_name\n",
    "}\n",
    "\n",
    "list(model.class_to_idx.items())[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(model.idx_to_class.items())[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"model name we are using is: \", model.__class__.__name__)\n",
    "print(\"note that this name, does not include the version number - so not as useful as manually appended name above\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-02T16:51:40.439601Z",
     "start_time": "2019-01-02T16:51:40.434599Z"
    }
   },
   "source": [
    "### build network - loss & optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# ACTION: Build and train your network\n",
    "\n",
    "# move tensors to GPU if CUDA is available\n",
    "if train_on_gpu:\n",
    "    model = model.to('cuda')\n",
    "\n",
    "# use torch.summary\n",
    "summary(model, input_size = (3, image_size_required, image_size_required), batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-01-02T16:52:02.463296Z",
     "start_time": "2019-01-02T16:52:02.460798Z"
    }
   },
   "source": [
    "### train network"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "TEMP - change cell below to not run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model,\n",
    "          optimizer,\n",
    "          criterion,\n",
    "          train_loader,\n",
    "          valid_loader,\n",
    "          global_saved_file_name,\n",
    "          n_epochs,\n",
    "          ):\n",
    "    # number of epochs to train; defined in global constants @top\n",
    "    n_epochs = n_epochs\n",
    "    # setup parameters to stop training if no improvement\n",
    "    epochs_no_improve = 0\n",
    "    epochs_noprogress_limit = 10\n",
    "\n",
    "    valid_loss_min = np.Inf  # track change in validation loss\n",
    "\n",
    "    # all scheduler args defined in global constants\n",
    "    scheduler = MultiStepLR(optimizer,\n",
    "                            milestones,\n",
    "                            gamma)\n",
    "\n",
    "    # track history of results for plotting\n",
    "    history = []\n",
    "    \n",
    "    # to help saving state_dict, and work around different layer names in model\n",
    "    # configure variable to match final layer name (e.g. classifier or fc)\n",
    "    if model.classifier_layer_name == 'classifier':\n",
    "        classifier_layer = model.classifier\n",
    "    if model.classifier_layer_name == 'fc':\n",
    "        classifier_layer = model.fc\n",
    "    \n",
    "    status_message = (f\"Training started on { chosen_model } model; output to { global_saved_file_name }\")\n",
    "    slack_message(status_message, [], channel_id)\n",
    "    \n",
    "    for epoch in range(1, n_epochs + 1):\n",
    "\n",
    "        # check steps to learning rate\n",
    "        scheduler.step()\n",
    "\n",
    "        # keep track of LOSS and ACCURACY for both training and validation\n",
    "        train_loss = 0\n",
    "        valid_loss = 0\n",
    "\n",
    "        train_acc = 0\n",
    "        valid_acc = 0\n",
    "\n",
    "        ###################\n",
    "        # train the model #\n",
    "        ###################\n",
    "        model.train()\n",
    "        for data, target in train_loader:\n",
    "            # move tensors to GPU if CUDA is available\n",
    "            if train_on_gpu:\n",
    "                data, target = data.cuda(), target.cuda()\n",
    "                model = model.to('cuda')\n",
    "                \n",
    "            # clear the gradients of all optimized variables\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # forward pass: compute predicted outputs by passing inputs to the model\n",
    "\n",
    "            # predicted outputs will be log probabilities\n",
    "            output = model(data)\n",
    "\n",
    "            # if inception_v3 use: output, aux = model(data)\n",
    "\n",
    "            # calculate the batch loss\n",
    "            loss = criterion(output, target)\n",
    "            \n",
    "            # backward pass\n",
    "            # compute gradient of the loss; then update parameters\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            # update training loss\n",
    "            train_loss += loss.item() * data.size(0)\n",
    "\n",
    "            # Estimate accuracy by finding max log probability\n",
    "            _, prediction = torch.max(output, dim=1)\n",
    "            correct_tensor = prediction.eq(target.data.view_as(prediction))\n",
    "\n",
    "            # Must convert correct_tensor from int to float to average\n",
    "            accuracy = torch.mean(correct_tensor.type(torch.FloatTensor))\n",
    "\n",
    "            # Multiply average accuracy times the number of examples in batch\n",
    "            train_acc += accuracy.item() * data.size(0)\n",
    "\n",
    "        ######################\n",
    "        # validate the model #\n",
    "        ######################\n",
    "        model.eval()\n",
    "        for data, target in valid_loader:\n",
    "            # move tensors to GPU if CUDA is available\n",
    "            if train_on_gpu:\n",
    "                data, target = data.cuda(), target.cuda(), \n",
    "                model = model.to('cuda')\n",
    "\n",
    "            # forward pass: compute predicted outputs by passing inputs to the model\n",
    "            # IMPORTANT - because we are using inceptionv3, the results are returned as a tuple that has to be unpacked per\n",
    "            # https://github.com/pytorch/vision/issues/302\n",
    "            # or we will get this error:  `AttributeError: 'tuple' object has no attribute 'log_softmax'`\n",
    "            # , aux\n",
    "\n",
    "            output = model(data)\n",
    "\n",
    "            # calculate the batch loss\n",
    "            loss = criterion(output, target)\n",
    "\n",
    "            # update average validation loss\n",
    "            valid_loss += loss.item() * data.size(0)\n",
    "\n",
    "            # Calculate validation accuracy\n",
    "            _, prediction = torch.max(output, dim=1)\n",
    "            correct_tensor = prediction.eq(target.data.view_as(prediction))\n",
    "            accuracy = torch.mean(correct_tensor.type(torch.FloatTensor))\n",
    "\n",
    "            # Multiply average accuracy times the number of examples\n",
    "            valid_acc += accuracy.item() * data.size(0)\n",
    "\n",
    "        # calculate average losses\n",
    "        train_loss = train_loss / len(train_loader.dataset)\n",
    "        valid_loss = valid_loss / len(valid_loader.dataset)\n",
    "\n",
    "        # Calculate average accuracy\n",
    "        train_acc = train_acc / len(train_loader.dataset)\n",
    "        valid_acc = valid_acc / len(valid_loader.dataset)\n",
    "\n",
    "        # print training/validation statistics\n",
    "        print(f\"epoch: {epoch}\\t\"\n",
    "              f\"Training Loss:  {train_loss:.6}\\t\\t\"\n",
    "              f\"Validation Loss:  {valid_loss:.6}\\n\\t\\t\"\n",
    "              f\"Training Accuracy:  {train_acc:.6}\\t\\t\"\n",
    "              f\"Validation Accuracy:  {valid_acc:.6}\"\n",
    "              )\n",
    "\n",
    "        history.append([epoch, train_loss, valid_loss, train_acc, valid_acc])\n",
    "\n",
    "        # save model if validation loss has decreased\n",
    "        if valid_loss <= valid_loss_min:\n",
    "\n",
    "            print(\n",
    "                f\"\\t\\tValidation loss decreased {valid_loss_min:.6} --> {valid_loss:.6} saving model...\\n\")\n",
    "            current_state = {'model': model,\n",
    "                             'model_name': model.model_name,\n",
    "                             'state_dict': model.state_dict(),\n",
    "                             'optimizer': optimizer.state_dict(),\n",
    "                             'classifier': classifier_layer,  # can be model.classifier or model.fc\n",
    "                             'epoch': epoch + 1,\n",
    "                             'batch_size': batch_size,\n",
    "                             'class_to_idx': model.class_to_idx,\n",
    "                             'idx_to_class': model.idx_to_class\n",
    "                             }\n",
    "\n",
    "            # attach the optimizer to our model\n",
    "            model.optimizer = optimizer\n",
    "            model.epoch = epoch\n",
    "\n",
    "            torch.save(current_state, global_saved_file_name)\n",
    "\n",
    "            valid_loss_min = valid_loss\n",
    "            \n",
    "        else:\n",
    "            epochs_no_improve += 1\n",
    "            # Check early stopping condition\n",
    "            if epochs_no_improve == epochs_noprogress_limit:\n",
    "                stopping_message = (\n",
    "                    f\"Early stopping - \"\n",
    "                    f\"training has not improved in \"\n",
    "                    f\"{ epochs_noprogress_limit } epochs \\n\")\n",
    "                print(stopping_message)\n",
    "                slack_message(stopping_message, [], channel_id)\n",
    "\n",
    "                # Load in the best model\n",
    "                model = torch.load(global_saved_file_name)\n",
    "                break\n",
    "            else:\n",
    "                continue\n",
    "\n",
    "    print(\"\\n ##########################################################\")\n",
    "\n",
    "    # dump training history to file\n",
    "    history = pd.DataFrame(history, columns=['epoch',\n",
    "                                             'train_loss',\n",
    "                                             'valid_loss',\n",
    "                                             'train_acc',\n",
    "                                             'valid_acc'])\n",
    "    \n",
    "    history_filename = (global_saved_file_name + \".history.csv\")\n",
    "    history.to_csv(os.path.join(data_dir, history_filename))\n",
    "    print(\"history.csv file saved\")\n",
    "    \n",
    "    # post status to slack\n",
    "    status_message = (f\"epoch: { epoch } validation loss better; valid_acc = {valid_acc:.6} saving\")\n",
    "    slack_message(status_message, [], channel_id)\n",
    "    status_message = (f\"{ global_saved_file_name}history.v60.csv file saved\")\n",
    "    slack_message(status_message, [], channel_id)\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# switch to RAWNB convert to skip retraining on cpu only platforms\n",
    "train(model,\n",
    "      optimizer,\n",
    "      criterion,\n",
    "      train_loader,\n",
    "      valid_loader,\n",
    "      global_saved_file_name,\n",
    "      n_epochs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the checkpoint\n",
    "\n",
    "At this point it's good to write a function that can load a checkpoint and rebuild the model. That way you can come back to this project and keep working on it without having to retrain the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Write a function that loads a checkpoint and rebuilds the model\n",
    "\n",
    "\n",
    "# this function loads pretrained model AND state\n",
    "def load_checkpoint(global_saved_file_name):\n",
    "    \"\"\"\n",
    "    Loads previously trained neural network\n",
    "    :param global_saved_file_name: name of checkpoint\n",
    "    \n",
    "    :return model ready for inference based on previous training\n",
    "    \"\"\"\n",
    "    filename = global_saved_file_name\n",
    "    print(\"\\n LOADING checkpoint:\")\n",
    "    # Note: Input model & optimizer should be pre-defined.  This routine only updates their states.\n",
    "\n",
    "    \n",
    "    def confirm_checkpoint_exists(filename):\n",
    "        if not os.path.isfile(filename):\n",
    "            print(\"=> no checkpoint found at '{}'\".format(filename))\n",
    "            raise ValueError(\"Checkpoint file not found\")\n",
    "        else:\n",
    "            return True\n",
    "        \n",
    "    \n",
    "    def unpack_state_dict(checkpoint):\n",
    "        \"\"\"\n",
    "        pytorch state_dict is complicated\n",
    "        this function unpacks the dictionary and loads the values (state) into the model\n",
    "        :param checkpoint: model that has been loaded into pytorch\n",
    "        \"\"\"\n",
    "        if 'state_dict' not in checkpoint:\n",
    "            raise ValueError(\"No `state_dict` exists in the saved checkpoint file\")\n",
    "        \n",
    "        else:\n",
    "            print(\"`state_dict` exists in checkpoint\")\n",
    "\n",
    "            print(\"iterating through state_dict keys\")\n",
    "            # state_dict = checkpoint['state_dict']\n",
    "            unpacked_state_dict = OrderedDict()\n",
    "            for k, v in checkpoint['state_dict'].items():\n",
    "                name = k[7:]  # remove 'module.' of dataparallel\n",
    "                unpacked_state_dict[name] = v\n",
    "\n",
    "            model.load_state_dict(unpacked_state_dict, \n",
    "                                  strict=False)         # strict False required\n",
    "        return True\n",
    "\n",
    "    confirm_checkpoint_exists(global_saved_file_name)\n",
    "    \n",
    "    # load model and optimizer state\n",
    "    print(\"=> loading checkpoint '{}' \\n\".format(filename))\n",
    "    checkpoint = torch.load(filename, map_location=device)\n",
    "    print(checkpoint.keys())\n",
    "\n",
    "    model = checkpoint['model']\n",
    "    \n",
    "    # unpack nested state using function & then load it\n",
    "    unpack_state_dict(checkpoint)\n",
    "    model.load_state_dict(checkpoint['state_dict'])\n",
    "    \n",
    "    # load remaining model keys\n",
    "    model.model_name = checkpoint['model_name']\n",
    "    model.optimizer = checkpoint['optimizer']\n",
    "    model.classifier = checkpoint['classifier']    \n",
    "    model.class_to_idx = checkpoint['class_to_idx']\n",
    "    model.idx_to_class = checkpoint['idx_to_class']\n",
    "    model.epoch = checkpoint['epoch']\n",
    "    model.batch_size = checkpoint['batch_size']\n",
    "    \n",
    "    # freeze all parameters - we're doing inference next\n",
    "    for param in model.parameters():\n",
    "        param.requires_grad = False\n",
    "\n",
    "    print(\"model_state loaded\")\n",
    "\n",
    "    return model\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = load_checkpoint(global_saved_file_name)\n",
    "\n",
    "try:\n",
    "   model.eval()\n",
    "   print(\"model in evaluation mode\")\n",
    "except AttributeError as error:\n",
    "   print(error)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference for classification\n",
    "\n",
    "Now you'll write a function to use a trained network for inference. That is, you'll pass an image into the network and predict the class of the flower in the image. Write a function called `predict` that takes an image and a model, then returns the top $K$ most likely classes along with the probabilities. It should look like \n",
    "\n",
    "```python\n",
    "probs, classes = predict(image_path, model)\n",
    "print(probs)\n",
    "print(classes)\n",
    "> [ 0.01558163  0.01541934  0.01452626  0.01443549  0.01407339]\n",
    "> ['70', '3', '45', '62', '55']\n",
    "```\n",
    "\n",
    "First you'll need to handle processing the input image such that it can be used in your network. \n",
    "\n",
    "## Image Preprocessing\n",
    "\n",
    "You'll want to use `PIL` to load the image ([documentation](https://pillow.readthedocs.io/en/latest/reference/Image.html)). It's best to write a function that preprocesses the image so it can be used as input for the model. This function should process the images in the same manner used for training. \n",
    "\n",
    "First, resize the images where the shortest side is 256 pixels, keeping the aspect ratio. This can be done with the [`thumbnail`](http://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.thumbnail) or [`resize`](http://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.thumbnail) methods. Then you'll need to crop out the center 224x224 portion of the image.\n",
    "\n",
    "Color channels of images are typically encoded as integers 0-255, but the model expected floats 0-1. You'll need to convert the values. It's easiest with a Numpy array, which you can get from a PIL image like so `np_image = np.array(pil_image)`.\n",
    "\n",
    "As before, the network expects the images to be normalized in a specific way. For the means, it's `[0.485, 0.456, 0.406]` and for the standard deviations `[0.229, 0.224, 0.225]`. You'll want to subtract the means from each color channel, then divide by the standard deviation. \n",
    "\n",
    "And finally, PyTorch expects the color channel to be the first dimension but it's the third dimension in the PIL image and Numpy array. You can reorder dimensions using [`ndarray.transpose`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.transpose.html). The color channel needs to be first and retain the order of the other two dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check your work, the function below converts a PyTorch tensor and displays it in the notebook. If your `process_image` function works, running the output through this function should return the original image (except for the cropped out portions)."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-01T01:04:54.414871Z",
     "start_time": "2019-02-01T01:04:54.409867Z"
    }
   },
   "source": [
    "# original manual image cropping; messy - there must be a better way\n",
    "\n",
    "def process_image(image_path):\n",
    "    \"\"\"\n",
    "    Convert an image path into a PyTorch tensor\n",
    "    :param image_path: path to the image\n",
    "    \n",
    "    :return image as a pytorch tensor\n",
    "    \"\"\"\n",
    "\n",
    "    image = Image.open(image_path)\n",
    "    # Resize\n",
    "    image = image.resize((256, 256))\n",
    "\n",
    "    # Center crop\n",
    "    width = 256\n",
    "    height = 256\n",
    "    new_width = 224\n",
    "    new_height = 224\n",
    "\n",
    "    left = (width - new_width) / 2\n",
    "    top = (height - new_height) / 2\n",
    "    right = (width + new_width) / 2\n",
    "    bottom = (height + new_height) / 2\n",
    "    image = image.crop((left, top, right, bottom))\n",
    "\n",
    "    # Convert to numpy, transpose color dimension and normalize\n",
    "    # img = np.array(img).transpose((2, 0, 1)) / 256\n",
    "\n",
    "    # Standardization\n",
    "    # means = np.array([0.485, 0.456, 0.406]).reshape((3, 1, 1))\n",
    "    # stds = np.array([0.229, 0.224, 0.225]).reshape((3, 1, 1))\n",
    "\n",
    "    # image = image - means\n",
    "    # image = image / stds\n",
    "\n",
    "    image = np.transpose(image, (2, 0, 1))\n",
    "    \n",
    "    # normalize images based on training data std and mean (given above)\n",
    "    # .reshape(3, 1, 1)\n",
    "    # std_train_imgs = np.array(std_train_imgs).reshape((3, 1, 1))\n",
    "    print(image.shape)\n",
    "    \n",
    "    image = std_train_imgs * image + mean_train_imgs\n",
    "    \n",
    "    # clip to range to suppress warnings\n",
    "    image = np.clip(image, 0, 1)\n",
    "    \n",
    "    image_tensor = torch.Tensor(image)\n",
    "\n",
    "    return image_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_image(process_image_path):\n",
    "    \"\"\"\n",
    "    Prepares a PIL image for our pytorch model\n",
    "    Applies same scaling, crop, normalization as training images\n",
    "    Uses pytorch transforms instead of manual cropping - easier to follow\n",
    "    \n",
    "    :param image_path: path to PIL image\n",
    "    :type image_path: string\n",
    "    \n",
    "    :return tensor array representing the image\n",
    "    \"\"\"\n",
    "    \n",
    "    # Process a PIL image for use in a PyTorch model\n",
    "    # tensor.numpy().transpose(1, 2, 0)    \n",
    "    if not os.path.isfile(process_image_path) :\n",
    "        print(\"path error in process image function\")\n",
    "    \n",
    "    image = Image.open(process_image_path)\n",
    "    \n",
    "    # use pytorch transforms, much cleaner\n",
    "    preprocess = transforms.Compose([\n",
    "                        transforms.Resize(256),\n",
    "                        transforms.CenterCrop(size=image_size_required),\n",
    "                        transforms.ToTensor(),\n",
    "                        transforms.Normalize(mean_train_imgs, std_train_imgs)\n",
    "                        ])\n",
    "    \n",
    "    image_tensor = preprocess(image)  # note transformat.ToTensor() step\n",
    "    \n",
    "    return image_tensor\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def imageshow_tensor(image_tensor, ax=None, title=None):\n",
    "    \"\"\"\n",
    "    Display an image passed in as a tensor\n",
    "    :param image_tesnsor: tensor representing and image; \n",
    "    channels in order of (color, height, width)\n",
    "    : return display and return matplotlib object, and PIL image\n",
    "    \"\"\"\n",
    "\n",
    "    if ax is None:\n",
    "        fig, ax = plt.subplots()\n",
    "\n",
    "    # Set the color channel as the third dimension\n",
    "    image = image_tensor.numpy().transpose((1, 2, 0))\n",
    "\n",
    "    # Reverse the preprocessing steps\n",
    "    mean = np.array([0.485, 0.456, 0.406])\n",
    "    std = np.array([0.229, 0.224, 0.225])\n",
    "    image = std * image + mean\n",
    "\n",
    "    # Clip the image pixel values\n",
    "    image = np.clip(image, 0, 1)\n",
    "\n",
    "    ax.imshow(image)\n",
    "    plt.axis('off')\n",
    "\n",
    "    return ax, image\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_2 = process_image(valid_dir + \"/10/image_07094.jpg\")\n",
    "image_2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imageshow_tensor(image_2, ax=None);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class Prediction\n",
    "\n",
    "Once you can get images in the correct format, it's time to write a function for making predictions with your model. A common practice is to predict the top 5 or so (usually called top-$K$) most probable classes. You'll want to calculate the class probabilities then find the $K$ largest values.\n",
    "\n",
    "To get the top $K$ largest values in a tensor use [`x.topk(k)`](http://pytorch.org/docs/master/torch.html#torch.topk). This method returns both the highest `k` probabilities and the indices of those probabilities corresponding to the classes. You need to convert from these indices to the actual class labels using `class_to_idx` which hopefully you added to the model or from an `ImageFolder` you used to load the data ([see here](#Save-the-checkpoint)). Make sure to invert the dictionary so you get a mapping from index to class as well.\n",
    "\n",
    "Again, this method should take a path to an image and a model checkpoint, then return the probabilities and classes.\n",
    "\n",
    "```python\n",
    "probs, classes = predict(image_path, model)\n",
    "print(probs)\n",
    "print(classes)\n",
    "> [ 0.01558163  0.01541934  0.01452626  0.01443549  0.01407339]\n",
    "> ['70', '3', '45', '62', '55']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write function to predict class of input image\n",
    "from torch.autograd import Variable\n",
    "\n",
    "def predict(image_path, model, topk=5):\n",
    "    \"\"\"\n",
    "    Predict classification of an image using a trained model\n",
    "    :param image_path: path to image being tested\n",
    "    :type image_path: string reprsenting path to the image\n",
    "    :param model: pretrained neural network\n",
    "    :param topk: number of top predictions to return\n",
    "    :type topk: int\n",
    "    \n",
    "    :return \n",
    "        img_tensor.cpu().squeeze() - tensor of image\n",
    "        top_p: (list) float probabilities of top(int) classes in order\n",
    "        top_classes: (int) class numbers of top(int) classes, in order\n",
    "        real_class: (int) class number of the actual class\n",
    "        real_class_name: (str) name of the actual class\n",
    "    \"\"\"\n",
    "    \n",
    "    gc.collect()\n",
    "    \n",
    "    # mitigate path issues with different systems with os.sep\n",
    "    real_class = image_path.split(os.sep)[-2]\n",
    "\n",
    "    # Convert image to pytorch tensor\n",
    "    image_tensor = process_image(image_path)\n",
    "\n",
    "    if train_on_gpu:\n",
    "        image_tensor = image_tensor.view(1, 3, 224, 224).cuda()\n",
    "        model = model.to('cuda')\n",
    "    else:\n",
    "        image_tensor = image_tensor.view(1, 3, 224, 224)\n",
    "\n",
    "    # set model to eval mode\n",
    "    model.eval()\n",
    "\n",
    "    inputs = Variable(image_tensor)\n",
    "    logits = model.forward(inputs)\n",
    "    probabilities = F.softmax(logits, dim=1)\n",
    "\n",
    "    topk = probabilities.cpu().topk(topk)\n",
    "\n",
    "    real_class_name = ''.join([tup[1] for tup in list(model.idx_to_class.items()) if tup[0] == int(real_class)])\n",
    "\n",
    "\n",
    "    top_probabilities, classes = (e.data.numpy().squeeze().tolist() for e in topk)\n",
    "    top_p = top_probabilities\n",
    "    class_names = train_datasets.classes\n",
    "    top_classes = [str(cat) + str(\" : \") + dict(cat_to_name)[cat] for cat in classes]\n",
    "\n",
    "    # PRINT RESULTS FOR EACH VARIABLE\n",
    "    print(\"real class: \", real_class)\n",
    "    print(\"real class name: \", real_class_name)\n",
    "    print(\"top classes & flower names = \", top_classes)\n",
    "    print(\"top_p probabilities = \", top_p)\n",
    "\n",
    "    return image_tensor.cpu().squeeze(), real_class, real_class_name, top_classes, top_p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# another predict single image, manually selected\n",
    "image_3_path = \"47\\image_04957.jpg\"\n",
    "\n",
    "image_3_path = os.path.join(valid_dir, image_3_path)\n",
    "print(image_3_path)\n",
    "#image_3_path = \"flower_data/valid/47/image_04957.jpg\"\n",
    "print(os.path.exists(image_3_path))\n",
    "      \n",
    "predict(image_3_path, model, topk=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_random_image():\n",
    "    \"\"\"\n",
    "    Select and returns one random test image from the validation directory\n",
    "    use os.path.join to protect against os-specific path issues\n",
    "    :param none - no input, uses global vars for path\n",
    "    :return image_path: path to image (jpg, etc.)\n",
    "    \"\"\"\n",
    "    \n",
    "    dir_choice = str(np.random.choice(os.listdir(valid_dir)))\n",
    "    dir_path = os.path.join(valid_dir, dir_choice)\n",
    "    files = os.listdir(dir_path)\n",
    "    # select a random file from directory; protect vs empty range\n",
    "    if len(files) >= 2:\n",
    "        index = random.randint(1, (len(files)-1))  # start / end\n",
    "    else:\n",
    "        index = 0\n",
    "    \n",
    "    image_path = os.path.join(dir_path, files[index])\n",
    "    \n",
    "    # uncomment this section to look at details in this dir\n",
    "    # print(\"class = \", path.split(os.sep)[-1])\n",
    "    # print(\"number of files in class = \", len(files))\n",
    "    # print(\"index in class chosen = \", index)\n",
    "    # print(\"image path = \", image_path)\n",
    "    # print(\"image name = \", image_path.split(os.sep)[-1])\n",
    "    # for i in range(len(files)):\n",
    "    #    print(\"index chosen =\", len(files), \"  index = \", i, \"  image path=\", path + files[i])\n",
    "        \n",
    "    return image_path\n",
    "\n",
    "get_random_image();\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# another predict single image, manually selected\n",
    "image_3_path = \"47\\image_04957.jpg\"\n",
    "\n",
    "image_3_path = os.path.join(valid_dir, image_3_path)\n",
    "print(image_3_path)\n",
    "#image_3_path = \"flower_data/valid/47/image_04957.jpg\"\n",
    "print(os.path.exists(image_3_path))\n",
    "      \n",
    "predict(image_3_path, model, topk=5)\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-02T00:57:24.259777Z",
     "start_time": "2019-02-02T00:57:24.255779Z"
    }
   },
   "source": [
    "# confirm/fix os path issues\n",
    "image_3_path = \"47\\image_04957.jpg\"\n",
    "\n",
    "newpath = os.path.join(valid_dir, image_3_path)\n",
    "print(\"newestpath = \", newpath)\n",
    "print(\"newest exists: \", os.path.exists(newpath))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict single image - random\n",
    "\n",
    "predict(get_random_image(), model, topk=5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sanity Checking\n",
    "\n",
    "Now that you can use a trained model for predictions, check to make sure it makes sense. Even if the validation accuracy is high, it's always good to check that there aren't obvious bugs. Use `matplotlib` to plot the probabilities for the top 5 classes as a bar graph, along with the input image. It should look like this:\n",
    "\n",
    "<img src='assets/inference_example.png' width=300px>\n",
    "\n",
    "You can convert from the class integer encoding to actual flower names with the `cat_to_name.json` file (should have been loaded earlier in the notebook). To show a PyTorch tensor as an image, use the `imshow` function defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ACTION - cleanup\n",
    "# TODO: Display an image along with the top 5 classes\n",
    "# return img_tensor.cpu().squeeze(), real_class, real_class_name, top_classes, top_p\n",
    "\n",
    "def display_prediction(image_path, model, topk_range):\n",
    "    \"\"\"\n",
    "    Display image to test and classification predictions\n",
    "    :param image_path: path to image file in directory\n",
    "    :type image_path: str\n",
    "    :param model: trained CNN\n",
    "    :param topk_range: number of predictions to return (e.g. 5)\n",
    "    :type topk_range: int\n",
    "    \n",
    "    :return True:  all functions must have return\n",
    "    \"\"\"\n",
    "\n",
    "    # returned from predict(): img_tensor.cpu().squeeze(), top_p, top_classes, real_class\n",
    "    # Get predictions\n",
    "    img, real_class, real_class_name, top_classes, top_probabilities = predict(image_path, model, topk_range)\n",
    "    # Convert results to dataframe for plotting\n",
    "    print(\"type of top classes = \", type(top_classes), top_classes)\n",
    "    result = pd.DataFrame({'p': top_probabilities}, index=top_classes)\n",
    "    print(top_probabilities)\n",
    "    print(result)\n",
    "\n",
    "    # Show the image and a chart of top predictions (name & probabilities)\n",
    "    plt.figure(figsize=(16, 5))\n",
    "    ax = plt.subplot(1, 2, 1)\n",
    "    ax, img = imageshow_tensor(img, ax=ax)\n",
    "    \n",
    "    title_text = list([tup for tup in list(model.idx_to_class.items()) if tup[0] == int(real_class)])\n",
    "    # list(title_text)\n",
    "    # Set title to be the actual class\n",
    "    ax.set_title(f\"{title_text[0][0]} / {title_text[0][1]}\", size=20)\n",
    "\n",
    "    ax = plt.subplot(1, 2, 2)\n",
    "    # Plot a bar plot of predictions\n",
    "    result.sort_values('p')['p'].plot.barh(color='blue', edgecolor='k', ax=ax)\n",
    "    \n",
    "    plt.xlim(0, 1)\n",
    "    plt.xlabel('Predicted Probability')\n",
    "    plt.tight_layout()\n",
    "    \n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_prediction(get_random_image(), model, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for flower in range(1,21):\n",
    "    display_prediction(get_random_image(), model, 5)\n",
    "    print(\"\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}