sagemaker-pipelines/step-decorator/computer-vision-examples/computer-vision-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14d517ee-f2f6-44b4-afe8-0c26c57e039c",
   "metadata": {},
   "source": [
    "# Computer vision model building pipeline using the step decorator\n",
    "\n",
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "---\n",
    "\n",
    "We have introduced a low-code experience for data scientists to convert the Machine Learning (ML) development code into repeatable and reusable workflow steps of [Amazon SageMaker Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) using a [@step decorator](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-step-decorator.html). This sample notebook demonstrates how to build a computer vision pipeline using a combination of the @step decorator and other pipeline steps.\n",
    "\n",
    "Specifically, this notebook builds a pipeline which:\n",
    "1. Uses the @step decorator to augment a retail image dataset;\n",
    "2. Uses the [Tuning step](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep) to train and tune a model using SageMaker's [Object Detection algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/object-detection.html);\n",
    "3. Uses the [Model step](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep) to create a model object for the best-performing model;\n",
    "4. Uses the [Transform step](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TransformStep) to run the test set through the best-performing model;\n",
    "5. Uses the @step decorator to evaluate the results.\n",
    "\n",
    "The screenshot below shows an example of a successful execution of this pipeline.\n",
    "\n",
    "![An example of a successful execution of the pipeline](example-execution.PNG \"Pipeline execution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10084708-91d8-4e9c-9570-5109e8dbbf41",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "If you run the notebook from a local IDE, please follow the \"AWS CLI Prerequisites\" section of the [Set-up Amazon SageMaker Prerequisites](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-set-up.html#gs-cli-prereq) to set up AWS credentials.\n",
    "\n",
    "The code in this notebook requires certain libraries to run successfully. Run the cell below to install these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00bce44a-0998-46e3-9f95-65c97467f5f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r ./requirements.txt -U"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d711f06b-21c6-4fa3-a371-4a77e920a8b4",
   "metadata": {},
   "source": [
    "First, let's import the libraries we will need and set up some variables related to our data. \n",
    "\n",
    "Notice that a `config.yaml` file is included with this code sample. This enables us to set [default values](https://docs.aws.amazon.com/sagemaker/latest/dg/train-remote-decorator-config.html) for the SageMaker SDK. \n",
    "\n",
    "We also use [Pipeline Session](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.pipeline_context.PipelineSession) which allows us to manage SageMaker APIs and AWS services needed under Pipeline Context. \n",
    "\n",
    "Finally, we use [S3FS](https://fs-s3fs.readthedocs.io/en/latest/) to more easily interact with our files in Amazon S3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd15d1b2-db6a-4eac-8af6-f867831f5063",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "from datetime import datetime\n",
    "from pathlib import Path\n",
    "\n",
    "import imageio.v2 as imageio\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import boto3\n",
    "import sagemaker\n",
    "import s3fs\n",
    "from sagemaker.utils import unique_name_from_base\n",
    "from sagemaker.workflow.function_step import step\n",
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.model_step import ModelStep\n",
    "\n",
    "\n",
    "pipeline_session = sagemaker.workflow.pipeline_context.PipelineSession()\n",
    "bucket_name = pipeline_session.default_bucket()\n",
    "prefix_project = \"computer-vision-retail-pipeline-demo\"  # main project folder in S3\n",
    "prefix_dataset = \"dataset\"  # where our dataset will be located\n",
    "prefix_models = \"models\"  # where our trained model weights will be saved\n",
    "prefix_output = \"output\"  # where the predictions will be saved\n",
    "prefix_results = \"results\"  # where the evaluation results will be stored\n",
    "\n",
    "local_dataset_folder = \"dataset-full\"\n",
    "local_train_manifest = f\"{local_dataset_folder}/train.manifest\"\n",
    "local_valid_manifest = f\"{local_dataset_folder}/validation.manifest\"\n",
    "local_test_manifest = f\"{local_dataset_folder}/test.manifest\"\n",
    "\n",
    "class_names = [  # names of the 10 products that we will be trying to detect\n",
    "    \"flakes\",\n",
    "    \"mm\",\n",
    "    \"coke\",\n",
    "    \"spam\",\n",
    "    \"nutella\",\n",
    "    \"doritos\",\n",
    "    \"ritz\",\n",
    "    \"skittles\",\n",
    "    \"mountaindew\",\n",
    "    \"evian\",\n",
    "]\n",
    "manifest_attributes = [\"source-ref\", \"retail-object-labeling\"]\n",
    "\n",
    "os.environ[\"SAGEMAKER_USER_CONFIG_OVERRIDE\"] = os.getcwd()  # set path to config file\n",
    "sagemaker_role = sagemaker.get_execution_role()\n",
    "region = pipeline_session.boto_region_name\n",
    "s3_fs = s3fs.S3FileSystem()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d18f820e-0dff-4709-a4c9-3d9c26fa7d75",
   "metadata": {},
   "source": [
    "## Download and preprocess the dataset\n",
    "\n",
    "This sample notebook uses a dataset and some code from the [Computer vision for retail inventory workshop](https://github.com/aws-samples/computer-vision-retail-workshop). The dataset contains 95 images of a shelf with different combinations of 10 supermarket products. The images have already been labeled using [SageMaker Ground Truth](https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html) and the dataset has been split into three parts:\n",
    "\n",
    "* Training (66 images)\n",
    "* Validation (19 images)\n",
    "* Testing (10 images)\n",
    "\n",
    "The cell below downloads the dataset and unzips it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd0d30c-5f90-454d-9053-65ded7ac3c00",
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget https://github.com/aws-samples/computer-vision-retail-workshop/raw/main/dataset/dataset-full.zip --no-check-certificate\n",
    "! unzip dataset-full.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bac95e17-ae31-4740-9947-b6299296a334",
   "metadata": {},
   "source": [
    "Using [S3Fs](https://s3fs.readthedocs.io/en/latest/), a Pythonic file interface for S3, upload the dataset to the S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f7a723e-191f-4347-bd31-36816a939ff4",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_fs.put(\n",
    "    f\"./{local_dataset_folder}/\",\n",
    "    f\"{bucket_name}/{prefix_project}/{prefix_dataset}/\",\n",
    "    recursive=True,\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2377e2b8-5c80-4c15-a276-40c83b09e619",
   "metadata": {},
   "source": [
    "To demonstrate the types of images contained in the dataset, the code below loads and displays one image and prints the associated annotations. The annotations are bounding boxes for each product displayed on the shelf."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c9d0a62-052d-454b-98e3-df7acbd059ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(local_train_manifest) as file_handle:\n",
    "    lines = file_handle.readlines()\n",
    "\n",
    "line_dict = json.loads(lines[0])  # load the 1st line of the manifest file\n",
    "filename = str(Path(line_dict[\"source-ref\"]).name)\n",
    "\n",
    "image = imageio.imread(f\"{local_dataset_folder}/{filename}\")\n",
    "plt.imshow(image)\n",
    "plt.grid(False)\n",
    "plt.axis(True)\n",
    "plt.title(f\"{local_dataset_folder}/{filename}\")\n",
    "plt.show()\n",
    "\n",
    "print(json.dumps(line_dict, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95e5929e-2cf4-4529-bd00-ac7f02310894",
   "metadata": {},
   "source": [
    "Currently, the `source-ref` attribute in the annotations file contains only the file name for each image. However, `source-ref` needs to be the S3 location of the image, because the algorithm will load the data from S3. For more information about the format used for manifest files, see [the documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/augmented-manifest.html).\n",
    "\n",
    "The cell below defines a method which loads a manifest file and updates each `source-ref` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "974f7368-b928-40e2-967c-e59d26b8239c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_manifest_sourceref(original_file: str, updated_file: str, bucket_name: str, prefix: str):\n",
    "    new_manifest = []\n",
    "\n",
    "    with open(original_file) as read_file:\n",
    "        lines = read_file.readlines()\n",
    "\n",
    "    for line in lines:\n",
    "        annotation = json.loads(line)\n",
    "        source_ref = str(Path(annotation[\"source-ref\"]).name)\n",
    "        updated_source_ref = f\"s3://{bucket_name}/{prefix}/{source_ref}\"\n",
    "        annotation[\"source-ref\"] = updated_source_ref\n",
    "        new_manifest.append(json.dumps(annotation))\n",
    "\n",
    "    with open(updated_file, \"w\") as write_file:\n",
    "        for annotation in new_manifest:\n",
    "            write_file.write(f\"{annotation}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b11a9d9-9c5d-4787-b28f-8e195bede400",
   "metadata": {},
   "source": [
    "Later on, the test set will be used to test the model through a batch transform. For this, we need to remove the existing annotations and format it as documented in [S3DataSource](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_S3DataSource.html#SageMaker-Type-S3DataSource-S3DataType). The method below performs these operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c888cc3-6040-49e0-98a7-0e322f1f9b15",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_unlabeled_test_manifest(labeled_manifest: str, unlabeled_manifest: str):\n",
    "    new_manifest = [{\"prefix\": f\"s3://{bucket_name}/{prefix_project}/{prefix_dataset}/\"}]\n",
    "\n",
    "    with open(labeled_manifest) as read_file:\n",
    "        lines = read_file.readlines()\n",
    "\n",
    "    for line in lines:\n",
    "        annotation = json.loads(line)\n",
    "        source_ref = str(Path(annotation[\"source-ref\"]).name)\n",
    "        new_manifest.append(source_ref)\n",
    "\n",
    "    with open(unlabeled_manifest, \"w\") as write_file:\n",
    "        json.dump(new_manifest, write_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d76e1598-0808-47a0-9c1b-1fe11b150a9e",
   "metadata": {},
   "source": [
    "Let's set up the correct locations and file names for the manifest files with updated values for `source-ref`, their corresponding S3 URI, and the unlabeled test manifest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "539e11ca-ce38-4b79-bd83-2859777e600f",
   "metadata": {},
   "outputs": [],
   "source": [
    "updated_train_manifest = f\"{local_dataset_folder}/train-updated.manifest\"\n",
    "updated_valid_manifest = f\"{local_dataset_folder}/validation-updated.manifest\"\n",
    "updated_test_manifest = f\"{local_dataset_folder}/test-updated.manifest\"\n",
    "unlabeled_test_manifest = f\"{local_dataset_folder}/test-unlabeled.manifest\"\n",
    "\n",
    "s3_train_manifest = f\"s3://{bucket_name}/{prefix_project}/{prefix_dataset}/train-updated.manifest\"\n",
    "s3_valid_manifest = (\n",
    "    f\"s3://{bucket_name}/{prefix_project}/{prefix_dataset}/validation-updated.manifest\"\n",
    ")\n",
    "s3_test_manifest = f\"s3://{bucket_name}/{prefix_project}/{prefix_dataset}/test-updated.manifest\"\n",
    "s3_unlabeled_test = f\"s3://{bucket_name}/{prefix_project}/{prefix_dataset}/test-unlabeled.manifest\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27e72ce6-6cda-44ab-a1fc-5ad2598490be",
   "metadata": {},
   "source": [
    "Now run the previously defined method for each of the datasets (training, validation, testing) and upload the updated manifest files to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6f48498-c9f2-4d75-8b4b-40788c18a523",
   "metadata": {},
   "outputs": [],
   "source": [
    "update_manifest_sourceref(\n",
    "    local_train_manifest,\n",
    "    updated_train_manifest,\n",
    "    bucket_name,\n",
    "    f\"{prefix_project}/{prefix_dataset}\",\n",
    ")\n",
    "s3_fs.put_file(updated_train_manifest, s3_train_manifest)\n",
    "\n",
    "update_manifest_sourceref(\n",
    "    local_valid_manifest,\n",
    "    updated_valid_manifest,\n",
    "    bucket_name,\n",
    "    f\"{prefix_project}/{prefix_dataset}\",\n",
    ")\n",
    "s3_fs.put_file(updated_valid_manifest, s3_valid_manifest)\n",
    "\n",
    "update_manifest_sourceref(\n",
    "    local_test_manifest,\n",
    "    updated_test_manifest,\n",
    "    bucket_name,\n",
    "    f\"{prefix_project}/{prefix_dataset}\",\n",
    ")\n",
    "s3_fs.put_file(updated_test_manifest, s3_test_manifest)\n",
    "create_unlabeled_test_manifest(updated_test_manifest, unlabeled_test_manifest)\n",
    "s3_fs.put_file(unlabeled_test_manifest, s3_unlabeled_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f14757b-c646-47ff-9bdb-cd5fadfed933",
   "metadata": {},
   "source": [
    "## Augment dataset\n",
    "\n",
    "Currently, the dataset only contains 95 images split into three datasets. This leaves us with relatively few training images. Augmenting the dataset means we generate synthetic variations of the existing images to increase the size of the dataset. There are many types of augmentations which can be applied to images, including flipping, rotating, scaling, translation, adding noise, adjusting brightness, and more. For simplicity, the code below only flips each image horizontally, thereby doubling our training dataset.\n",
    "\n",
    "We define two methods: `augment_image` flips a single image and calculates the new bounding boxes for the annotations, and `augment_dataset` then applies this flipping logic to every image in the dataset. Notice how we use the @step decorator to turn `augment_dataset` into a step for the model building pipeline. It will automatically take the code and run it as a SageMaker training job when we execute the pipeline. The @step decorator has many [parameter options](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#step-decorator), but it also uses the defaults we specified in our SageMaker configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99092490-0c8c-484a-9937-ca1873df5dd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def augment_image(image_filename: str, bboxes: dict) -> tuple:\n",
    "    with s3_fs.open(image_filename) as image_handle:\n",
    "        # load image\n",
    "        image = imageio.imread(image_handle)\n",
    "        image_width = image.shape[1]\n",
    "\n",
    "        # flip image\n",
    "        image_flipped = image.copy()\n",
    "        image_flipped = np.fliplr(image_flipped)\n",
    "\n",
    "        # flip bounding boxes\n",
    "        bboxes_flipped = []\n",
    "        for bbox in bboxes:\n",
    "            bboxes_flipped.append(\n",
    "                {\n",
    "                    \"left\": image_width - (bbox[\"left\"] + bbox[\"width\"]),\n",
    "                    \"top\": bbox[\"top\"],\n",
    "                    \"width\": bbox[\"width\"],\n",
    "                    \"height\": bbox[\"height\"],\n",
    "                    \"class_id\": bbox[\"class_id\"],\n",
    "                }\n",
    "            )\n",
    "\n",
    "        return image_flipped, bboxes_flipped\n",
    "\n",
    "\n",
    "@step(name=\"AugmentData\", keep_alive_period_in_seconds=300)\n",
    "def augment_dataset(manifest_s3_path: str, output_path: str, output_manifest: str) -> tuple:\n",
    "    new_manifest = []\n",
    "\n",
    "    with s3_fs.open(manifest_s3_path) as input_handle:\n",
    "        lines = input_handle.readlines()\n",
    "\n",
    "        for line in lines:\n",
    "            line_dict = json.loads(line)  # load one json line (corresponding to one image)\n",
    "            filename_object = Path(line_dict[\"source-ref\"])\n",
    "            filename = str(filename_object.name)  # filename without the path\n",
    "\n",
    "            # add json line of the original image\n",
    "            new_manifest.append(json.dumps(line_dict))\n",
    "\n",
    "            # generate augmented images\n",
    "            print(\"Augmenting image:\", filename)\n",
    "            image_augm, bboxes_augm = augment_image(\n",
    "                image_filename=line_dict[\"source-ref\"],\n",
    "                bboxes=line_dict[\"retail-object-labeling\"][\"annotations\"],\n",
    "            )\n",
    "\n",
    "            # new image size of augmented image\n",
    "            image_height = image_augm.shape[0]\n",
    "            image_width = image_augm.shape[1]\n",
    "            if len(image_augm.shape) == 3:\n",
    "                image_depth = image_augm.shape[2]\n",
    "            else:\n",
    "                image_depth = 1\n",
    "            line_dict[\"retail-object-labeling\"][\"image_size\"] = [\n",
    "                {\"width\": image_width, \"height\": image_height, \"depth\": image_depth}\n",
    "            ]\n",
    "\n",
    "            # augmented image filename\n",
    "            filename_no_extension = str(filename_object.stem)  # filename without extension\n",
    "            filename_augmented = f\"{filename_no_extension}_augm.jpg\"\n",
    "            # image_augm_filename = f'augmented/{filename_augmented}'\n",
    "            imageio.imsave(filename_augmented, image_augm, quality=95)  # save locally\n",
    "            new_filename_s3 = f\"{output_path}/{filename_augmented}\"\n",
    "            line_dict[\"source-ref\"] = new_filename_s3  # add new filename to the manifest file\n",
    "            s3_fs.put_file(filename_augmented, new_filename_s3)\n",
    "\n",
    "            # new image bounding boxes\n",
    "            line_dict[\"retail-object-labeling\"][\"annotations\"] = bboxes_augm\n",
    "\n",
    "            # add a new json line for this augmentation image\n",
    "            new_manifest.append(json.dumps(line_dict))\n",
    "\n",
    "    augm_manifest_location = f\"{output_path}/{output_manifest}\"\n",
    "    with s3_fs.open(augm_manifest_location, \"w\") as output_handle:\n",
    "        for label in new_manifest:\n",
    "            output_handle.write(label + \"\\n\")\n",
    "\n",
    "    return augm_manifest_location, len(new_manifest)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e2c73fa-9077-4125-8586-eac7223b183c",
   "metadata": {},
   "source": [
    "The final trick to using the @step decorator is the delayed return. By calling our decorated `augment_dataset` method, SageMaker returns a `DelayedReturn` instance instead of running the function. A `DelayedReturn` instance is a proxy for the actual return of that function. The `DelayedReturn` instance can be passed to another function as an argument or directly to a pipeline instance as a step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71bce3e3-18da-4e85-b814-6b2c9569f077",
   "metadata": {},
   "outputs": [],
   "source": [
    "delayed_augment = augment_dataset(\n",
    "    s3_train_manifest,\n",
    "    f\"s3://{bucket_name}/{prefix_project}/{prefix_dataset}\",\n",
    "    \"train-augmented.manifest\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c3174ce-6295-47f7-a713-5842a7051f16",
   "metadata": {},
   "source": [
    "## Tune a model\n",
    "\n",
    "Next, let's set up an automatic hyperparameter tuning step in our pipeline which uses the built-in SageMaker Object Detection algorithm to train models. Since we are using a built-in algorithm, we don't have to write the algorithm code ourselves, and it makes sense to use the [Tuning step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning) without a decorator.\n",
    "\n",
    "First, the code below sets up an Estimator object for the Object Detection algorithm. It fetches the correct training image, sets up the input channels for the training and validation datasets, and specifies static hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b211699-af13-46ad-bbf7-41d7e3bbdf5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_channel = sagemaker.inputs.TrainingInput(\n",
    "    delayed_augment[0],\n",
    "    distribution=\"FullyReplicated\",\n",
    "    content_type=\"application/x-recordio\",\n",
    "    s3_data_type=\"AugmentedManifestFile\",\n",
    "    attribute_names=manifest_attributes,\n",
    "    record_wrapping=\"RecordIO\",\n",
    "    shuffle_config=sagemaker.inputs.ShuffleConfig(seed=1),\n",
    ")\n",
    "\n",
    "validation_channel = sagemaker.inputs.TrainingInput(\n",
    "    s3_valid_manifest,\n",
    "    distribution=\"FullyReplicated\",\n",
    "    content_type=\"application/x-recordio\",\n",
    "    record_wrapping=\"RecordIO\",\n",
    "    s3_data_type=\"AugmentedManifestFile\",\n",
    "    attribute_names=manifest_attributes,\n",
    ")\n",
    "\n",
    "training_image = sagemaker.image_uris.retrieve(\n",
    "    region=region, framework=\"object-detection\", version=\"1\", image_scope=\"training\"\n",
    ")\n",
    "\n",
    "estimator = sagemaker.estimator.Estimator(\n",
    "    training_image,\n",
    "    sagemaker_role,\n",
    "    input_mode=\"Pipe\",\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.p3.2xlarge\",\n",
    "    volume_size=50,\n",
    "    max_run=10 * 60 * 60,\n",
    "    output_path=f\"s3://{bucket_name}/{prefix_project}/{prefix_models}\",\n",
    "    sagemaker_session=pipeline_session,\n",
    ")\n",
    "\n",
    "estimator.set_hyperparameters(\n",
    "    base_network=\"resnet-50\",\n",
    "    use_pretrained_model=1,\n",
    "    early_stopping=True,\n",
    "    num_classes=len(class_names),\n",
    "    optimizer=\"adam\",\n",
    "    image_shape=512,\n",
    "    num_training_samples=delayed_augment[1],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f8e1812-b279-446c-8d88-36efbf07d82d",
   "metadata": {},
   "source": [
    "The code below sets up the [Hyperparameter Tuner](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html) to find the optimal values for `learning_rate` and `mini_batch_size`. \n",
    "\n",
    "Note that we are using a `ml.p3.2xlarge` instance for training. The Object Detection algorithm requires [GPU instances for training](https://docs.aws.amazon.com/sagemaker/latest/dg/object-detection.html#object-detection-instances). If you encounter any quota errors, please [request a quota increase](https://docs.aws.amazon.com/sagemaker/latest/dg/canvas-requesting-quota-increases.html) and then try again. If you have access to multiple instances of this type on your account, you can increase the `max_parallel_jobs` parameter below to speed up the tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ad5ce7d-3548-4751-961a-595d467d8e6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner = sagemaker.tuner.HyperparameterTuner(\n",
    "    estimator,\n",
    "    \"validation:mAP\",\n",
    "    objective_type=\"Maximize\",\n",
    "    base_tuning_job_name=\"retail-ODD-lightsaber\",\n",
    "    hyperparameter_ranges={\n",
    "        \"learning_rate\": sagemaker.tuner.ContinuousParameter(0.00001, 0.001),\n",
    "        \"mini_batch_size\": sagemaker.tuner.IntegerParameter(8, 16),\n",
    "    },\n",
    "    max_parallel_jobs=1,\n",
    "    max_jobs=5,\n",
    ")\n",
    "\n",
    "step_tuning_args = tuner.fit(\n",
    "    inputs={\"train\": train_channel, \"validation\": validation_channel},\n",
    ")\n",
    "\n",
    "step_tuning = sagemaker.workflow.steps.TuningStep(\n",
    "    name=\"TuneModel\",\n",
    "    step_args=step_tuning_args,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8544e531-61c3-4897-b1b6-e23f008e5afc",
   "metadata": {},
   "source": [
    "## Run best model on test set\n",
    "\n",
    "Once the tuning step has found the best performing model, we want to create this model (using [ModelStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep)) and use it with a [TransformStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TransformStep) to run it on the test set. The predictions will be uploaded onto Amazon S3.\n",
    "\n",
    "The code below creates the model object based on the best model from the tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20dc627a-e473-4af8-afec-8573761f2b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_image = sagemaker.image_uris.retrieve(\n",
    "    region=region, framework=\"object-detection\", version=\"1\", image_scope=\"inference\"\n",
    ")\n",
    "\n",
    "best_model = sagemaker.model.Model(\n",
    "    image_uri=inference_image,\n",
    "    model_data=step_tuning.get_top_model_s3_uri(\n",
    "        top_k=0, s3_bucket=bucket_name, prefix=f\"{prefix_project}/{prefix_models}\"\n",
    "    ),\n",
    "    sagemaker_session=pipeline_session,\n",
    "    role=sagemaker_role,\n",
    ")\n",
    "\n",
    "step_create_model = ModelStep(\n",
    "    name=\"CreateModel\",\n",
    "    step_args=best_model.create(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30597b56-6b98-43e8-8ff9-fb5bfb52af11",
   "metadata": {},
   "source": [
    "The code below uses a batch transform job to run the model over the test data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "450c492d-07e4-4a80-8a66-dc6dafa178d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = sagemaker.transformer.Transformer(\n",
    "    model_name=step_create_model.properties.ModelName,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.p3.2xlarge\",\n",
    "    output_path=f\"s3://{bucket_name}/{prefix_project}/{prefix_output}\",\n",
    "    sagemaker_session=pipeline_session,\n",
    ")\n",
    "\n",
    "step_transform_args = transformer.transform(\n",
    "    data=s3_unlabeled_test,\n",
    "    content_type=\"application/x-image\",\n",
    "    data_type=\"ManifestFile\",\n",
    ")\n",
    "\n",
    "step_transform = sagemaker.workflow.steps.TransformStep(\n",
    "    name=\"TestModel\",\n",
    "    step_args=step_transform_args,\n",
    "    depends_on=[step_create_model],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69fb058-16bc-4017-b795-2f5a95027d83",
   "metadata": {},
   "source": [
    "## Evaluate test results\n",
    "\n",
    "Finally, the @step decorator can be used to evaluate the performance of the model using the predictions generated on the test set. The code below calculates [Intersection Over Union (IOU)](https://en.wikipedia.org/wiki/Jaccard_index), a metric that describes the degree of overlap between two bounding boxes, to find the similarity between the predicted bounding boxes and the ground truth bounding boxes.\n",
    "\n",
    "Based on this, it calculates true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for every class. Precision, recall, and F1 score are also calculated per class and as a macro average. The results are saved in Amazon S3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c490b24e-f3bf-4db7-9b83-549b0ba419e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_iou(BBoxW1, BBoxH1, BBoxL1, BBoxT1, BBoxW2, BBoxH2, BBoxL2, BBoxT2):\n",
    "    # intersection over union in order to match bboxes\n",
    "    BBoxR1 = BBoxL1 + BBoxW1\n",
    "    BBoxB1 = BBoxT1 + BBoxH1\n",
    "    BBoxR2 = BBoxL2 + BBoxW2\n",
    "    BBoxB2 = BBoxT2 + BBoxH2\n",
    "\n",
    "    int_L = max(BBoxL1, BBoxL2)\n",
    "    int_R = min(BBoxR1, BBoxR2)\n",
    "    int_T = max(BBoxT1, BBoxT2)\n",
    "    int_B = min(BBoxB1, BBoxB2)\n",
    "    intersection_area = max(0, int_R - int_L) * max(0, int_B - int_T)\n",
    "\n",
    "    bbox1_area = (BBoxR1 - BBoxL1) * (BBoxB1 - BBoxT1)\n",
    "    bbox2_area = (BBoxR2 - BBoxL2) * (BBoxB2 - BBoxT2)\n",
    "\n",
    "    iou = intersection_area / (bbox1_area + bbox2_area - intersection_area)\n",
    "\n",
    "    return iou\n",
    "\n",
    "\n",
    "@step(name=\"EvaluateModel\")\n",
    "def evaluate(\n",
    "    class_names: list,\n",
    "    testset_folder: str,\n",
    "    test_manifest_file: str,\n",
    "    threshold_iou: float,\n",
    "    threshold_confidence: float,\n",
    "    output_folder: str,\n",
    "):\n",
    "    # initialize performance df\n",
    "    data = np.zeros([len(class_names), 6], dtype=float)\n",
    "    df_class_performance = pd.DataFrame(data=data, columns=[\"TP\", \"FP\", \"FN\", \"PR\", \"RE\", \"F1\"])\n",
    "    df_classes = pd.DataFrame(data=class_names, columns=[\"CLASS\"])\n",
    "    df_class_performance = pd.concat([df_classes, df_class_performance], axis=1)\n",
    "\n",
    "    # open manifest file\n",
    "    with s3_fs.open(test_manifest_file) as read_file:\n",
    "        print(f\"Evaluating {test_manifest_file}...\")\n",
    "        lines = read_file.readlines()\n",
    "\n",
    "    # go through each JSON line\n",
    "    for line in lines:\n",
    "        ls_annotations = []\n",
    "        image_info = json.loads(line)\n",
    "\n",
    "        # get image\n",
    "        filename = Path(image_info[\"source-ref\"])\n",
    "        print(f\"Analyzing image {str(filename.name)}...\")\n",
    "        filename_with_path = Path(*filename.parts[2:])\n",
    "\n",
    "        # get predictions from batch transform\n",
    "        predictions_file = f\"{testset_folder}/{str(filename.name)}.out\"\n",
    "        with s3_fs.open(predictions_file) as read_file:\n",
    "            file_content = read_file.readlines()\n",
    "        predictions = json.loads(file_content[0])[\"prediction\"]\n",
    "        df_predictions = pd.DataFrame(\n",
    "            predictions, columns=[\"class\", \"confidence\", \"xmin\", \"ymin\", \"xmax\", \"ymax\"]\n",
    "        )\n",
    "\n",
    "        # filter low confidence predictions\n",
    "        df_predictions = df_predictions.loc[df_predictions[\"confidence\"] >= threshold_confidence]\n",
    "\n",
    "        image_width = image_info[\"retail-object-labeling\"][\"image_size\"][0][\"width\"]\n",
    "        image_height = image_info[\"retail-object-labeling\"][\"image_size\"][0][\"height\"]\n",
    "        df_predictions.loc[:, \"xmin\"] *= image_width\n",
    "        df_predictions.loc[:, \"xmax\"] *= image_width\n",
    "        df_predictions.loc[:, \"ymin\"] *= image_height\n",
    "        df_predictions.loc[:, \"ymax\"] *= image_height\n",
    "        df_predictions.loc[:, [\"xmin\", \"xmax\", \"ymin\", \"ymax\"]] = df_predictions.loc[\n",
    "            :, [\"xmin\", \"xmax\", \"ymin\", \"ymax\"]\n",
    "        ].round(decimals=0)\n",
    "\n",
    "        # get annotations from manifest GT file\n",
    "        for index, annotation in enumerate(image_info[\"retail-object-labeling\"][\"annotations\"]):\n",
    "            ls_ground_truth = []\n",
    "            ls_ground_truth.append(\n",
    "                image_info[\"retail-object-labeling\"][\"annotations\"][index][\"class_id\"]\n",
    "            )\n",
    "            ls_ground_truth.append(\n",
    "                image_info[\"retail-object-labeling\"][\"annotations\"][index][\"top\"]\n",
    "            )\n",
    "            ls_ground_truth.append(\n",
    "                image_info[\"retail-object-labeling\"][\"annotations\"][index][\"left\"]\n",
    "            )\n",
    "            ls_ground_truth.append(\n",
    "                image_info[\"retail-object-labeling\"][\"annotations\"][index][\"height\"]\n",
    "            )\n",
    "            ls_ground_truth.append(\n",
    "                image_info[\"retail-object-labeling\"][\"annotations\"][index][\"width\"]\n",
    "            )\n",
    "            ls_annotations.append(ls_ground_truth)\n",
    "        df_annotations = pd.DataFrame(\n",
    "            data=ls_annotations, columns=[\"class_id\", \"top\", \"left\", \"height\", \"width\"]\n",
    "        )\n",
    "\n",
    "        # create IOU array\n",
    "        mat_iou = np.zeros((len(df_annotations), len(df_predictions)), dtype=float)\n",
    "        for prediction_index in range(len(df_predictions)):\n",
    "            for annotation_index in range(len(df_annotations)):\n",
    "                iou = get_iou(\n",
    "                    BBoxW1=df_predictions.loc[prediction_index, \"xmax\"]\n",
    "                    - df_predictions.loc[prediction_index, \"xmin\"],\n",
    "                    BBoxH1=df_predictions.loc[prediction_index, \"ymax\"]\n",
    "                    - df_predictions.loc[prediction_index, \"ymin\"],\n",
    "                    BBoxL1=df_predictions.loc[prediction_index, \"xmin\"],\n",
    "                    BBoxT1=df_predictions.loc[prediction_index, \"ymin\"],\n",
    "                    BBoxW2=df_annotations.loc[annotation_index, \"width\"],\n",
    "                    BBoxH2=df_annotations.loc[annotation_index, \"height\"],\n",
    "                    BBoxL2=df_annotations.loc[annotation_index, \"left\"],\n",
    "                    BBoxT2=df_annotations.loc[annotation_index, \"top\"],\n",
    "                )\n",
    "                mat_iou[annotation_index, prediction_index] = iou\n",
    "        mat_iou[mat_iou < threshold_iou] = 0  # binarize IOU array\n",
    "        mat_iou[mat_iou > 0] = 1\n",
    "\n",
    "        # analyzing IOU array\n",
    "        for annotation_index in range(len(df_annotations)):\n",
    "            class_id_annotation = int(df_annotations.loc[annotation_index, \"class_id\"])\n",
    "\n",
    "            if mat_iou[annotation_index, :].sum() == 0:  # if no matches for this annotation\n",
    "                df_class_performance.loc[class_id_annotation, \"FN\"] += 1\n",
    "\n",
    "            elif (\n",
    "                mat_iou[annotation_index, :].sum() == 1\n",
    "            ):  # if only one matching for this annotation\n",
    "                indx_nonzero = np.nonzero(mat_iou[annotation_index, :])[0][0]\n",
    "                class_id_detection = int(df_predictions.loc[indx_nonzero, \"class\"])\n",
    "\n",
    "                if class_id_annotation == class_id_detection:\n",
    "                    df_class_performance.loc[class_id_detection, \"TP\"] += 1\n",
    "                else:\n",
    "                    df_class_performance.loc[class_id_detection, \"FP\"] += 1\n",
    "\n",
    "            elif (\n",
    "                mat_iou[annotation_index, :].sum() > 1\n",
    "            ):  # if more than one matching for this annotation\n",
    "                indx_nonzero = np.squeeze(\n",
    "                    np.nonzero(mat_iou[annotation_index, :])[0]\n",
    "                )  # many indices of nonzero ious\n",
    "                conf_detection = df_predictions.loc[\n",
    "                    indx_nonzero, \"confidence\"\n",
    "                ]  # many confidences of nonzero ious\n",
    "                indx_maxconf = indx_nonzero[\n",
    "                    np.argmax(conf_detection)\n",
    "                ]  # find the indx of max confidence\n",
    "                class_id_maxconf = df_predictions.loc[\n",
    "                    indx_maxconf, \"class\"\n",
    "                ]  # find the class of max confidence\n",
    "\n",
    "                indx_lowconf = np.delete(\n",
    "                    indx_nonzero, np.argmax(conf_detection)\n",
    "                )  # keep all the indexes without the one of max confidence\n",
    "                class_id_lowconf = df_predictions.loc[\n",
    "                    indx_lowconf, \"class\"\n",
    "                ]  # find the classes of the rest\n",
    "\n",
    "                if class_id_annotation == class_id_maxconf:\n",
    "                    df_class_performance.loc[\n",
    "                        class_id_maxconf, \"TP\"\n",
    "                    ] += 1  # the max confidence is TP\n",
    "                    df_class_performance.loc[class_id_lowconf, \"FP\"] += 1  # the rest are FP\n",
    "                else:\n",
    "                    df_class_performance.loc[class_id_maxconf, \"FP\"] += 1  # all are FP\n",
    "                    df_class_performance.loc[class_id_lowconf, \"FP\"] += 1\n",
    "\n",
    "            else:\n",
    "                print(\"Problem with negative IOU values!\")\n",
    "\n",
    "        for prediction_index in range(len(df_predictions)):\n",
    "            class_id_detection = int(df_predictions.loc[prediction_index, \"class\"])\n",
    "            if mat_iou[:, prediction_index].sum() == 0:  # if no matches for this detection\n",
    "                df_class_performance.loc[class_id_detection, \"FP\"] += 1\n",
    "\n",
    "    # estimate metrics per class\n",
    "    df_class_performance.loc[:, \"PR\"] = df_class_performance.loc[:, \"TP\"] / (\n",
    "        df_class_performance.loc[:, \"TP\"] + df_class_performance.loc[:, \"FP\"]\n",
    "    )\n",
    "    df_class_performance.loc[:, \"RE\"] = df_class_performance.loc[:, \"TP\"] / (\n",
    "        df_class_performance.loc[:, \"TP\"] + df_class_performance.loc[:, \"FN\"]\n",
    "    )\n",
    "    df_class_performance.loc[:, \"F1\"] = (\n",
    "        2 * df_class_performance.loc[:, \"PR\"] * df_class_performance.loc[:, \"RE\"]\n",
    "    ) / (df_class_performance.loc[:, \"PR\"] + df_class_performance.loc[:, \"RE\"])\n",
    "\n",
    "    mean_macro = [\n",
    "        \"macro Average\",\n",
    "        \"\",\n",
    "        \"\",\n",
    "        \"\",\n",
    "        df_class_performance.loc[:, \"PR\"].mean(),\n",
    "        df_class_performance.loc[:, \"RE\"].mean(),\n",
    "        df_class_performance.loc[:, \"F1\"].mean(),\n",
    "    ]\n",
    "    df_mean = pd.DataFrame(data=[mean_macro], columns=[\"CLASS\", \"TP\", \"FP\", \"FN\", \"PR\", \"RE\", \"F1\"])\n",
    "    df_class_performance = pd.concat([df_class_performance, df_mean], ignore_index=True)\n",
    "\n",
    "    output_file = f\"{output_folder}/results.csv\"\n",
    "    with s3_fs.open(output_file, \"w\") as output_handle:\n",
    "        df_class_performance.to_csv(output_handle)\n",
    "\n",
    "    return output_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae5316c-4a87-4cf8-bec2-32f5de5ef6cb",
   "metadata": {},
   "source": [
    "Once again, use a delayed return to create the step for the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58fa7836-fa58-47c1-bff1-f77d3960e578",
   "metadata": {},
   "outputs": [],
   "source": [
    "delayed_evaluate = evaluate(\n",
    "    class_names,\n",
    "    step_transform.properties.TransformOutput.S3OutputPath,\n",
    "    s3_test_manifest,\n",
    "    0.5,\n",
    "    0.2,\n",
    "    f\"s3://{bucket_name}/{prefix_project}/{prefix_results}\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b948cb33-f2b0-431a-a7c9-d94305d88233",
   "metadata": {},
   "source": [
    "## Build pipeline\n",
    "\n",
    "Finally, define the pipeline. You only have to provide the last step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc92f560-163e-4a1a-91d0-a01f06c7b43e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline(\n",
    "    name=unique_name_from_base(\"retail-computer-vision\"),\n",
    "    steps=[delayed_evaluate],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ceafe2-d792-4f58-9b44-63a85ca7cd6b",
   "metadata": {},
   "source": [
    "Create or update the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f05ce4a7-9c90-4b17-96dd-766f2a507e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.upsert(role_arn=sagemaker_role)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e1408c-ddc8-4002-a4eb-1ddc65fb83b0",
   "metadata": {},
   "source": [
    "Now run a pipeline execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46881c79-8051-4314-92e7-502113e8675b",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution = pipeline.start()\n",
    "execution.wait(delay=300, max_attempts=18)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c30be066-4ff1-4e8a-acfe-b32948eaec7f",
   "metadata": {},
   "source": [
    "You can view the execution's progress in SageMaker Studio, or you can run the cells below to retrieve information on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa927842-3cc4-45d0-ba6e-16b34444ef87",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28d0ff5c-2c1d-49e6-b888-b05d171c7020",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.list_steps()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85339096-3a8b-4d56-9d4f-3d81e1def31c",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file_s3_uri = execution.result(step_name=\"EvaluateModel\")\n",
    "print(output_file_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9612a6f0",
   "metadata": {},
   "source": [
    "When the pipeline execution has successfully finished, you will find a `results.csv` file in your S3 bucket. The full path to the results file is printed by the cell above. Below is an example of the output it will contain, but remember that your output will differ.\n",
    "\n",
    "|    | CLASS         | TP   | FP  | FN  | PR                 | RE                 | F1                 |\n",
    "|----|---------------|------|-----|-----|--------------------|--------------------|--------------------|\n",
    "| 0  | flakes        | 6.0  | 0.0 | 1.0 | 1.0                | 0.8571428571428571 | 0.923076923076923  |\n",
    "| 1  | mm            | 10.0 | 3.0 | 1.0 | 0.7692307692307693 | 0.9090909090909091 | 0.8333333333333333 |\n",
    "| 2  | coke          | 9.0  | 0.0 | 3.0 | 1.0                | 0.75               | 0.8571428571428571 |\n",
    "| 3  | spam          | 8.0  | 1.0 | 3.0 | 0.8888888888888888 | 0.7272727272727273 | 0.7999999999999999 |\n",
    "| 4  | nutella       | 13.0 | 0.0 | 1.0 | 1.0                | 0.9285714285714286 | 0.962962962962963  |\n",
    "| 5  | doritos       | 10.0 | 0.0 | 2.0 | 1.0                | 0.8333333333333334 | 0.9090909090909091 |\n",
    "| 6  | ritz          | 11.0 | 1.0 | 1.0 | 0.9166666666666666 | 0.9166666666666666 | 0.9166666666666666 |\n",
    "| 7  | skittles      | 10.0 | 0.0 | 0.0 | 1.0                | 1.0                | 1.0                |\n",
    "| 8  | mountaindew   | 13.0 | 0.0 | 1.0 | 1.0                | 0.9285714285714286 | 0.962962962962963  |\n",
    "| 9  | evian         | 5.0  | 0.0 | 2.0 | 1.0                | 0.7142857142857143 | 0.8333333333333333 |\n",
    "| 10 | macro Average |      |     |     | 0.9574786324786324 | 0.8564935064935064 | 0.899856994856995  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ea6e1d0-37fe-4a32-a961-ab23ed590b4a",
   "metadata": {},
   "source": [
    "## Clean Up\n",
    "\n",
    "This section helps clean up the resources created by running this notebook. \n",
    "\n",
    "<div class=\"alert alert-block alert-warning\"><b>Warning:</b> This section does not clean up the data and results uploaded to Amazon S3. Please delete these resources manually.</div>\n",
    "\n",
    "The code below deletes the SageMaker pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c184032c-d4dc-4cf6-93ad-978bae669dad",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a61f3b2",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker-pipelines|step-decorator|computer-vision-examples|computer-vision-pipeline.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}