How to deploy OpenVINO™ Model Server automatically onto Azure VM for OVaaS

This instruction shows how to launch Azure VM and deploy OpenVINO™ Model Server and specific model (IR format) on that from Azure CLI. With this instructions and scripts, We can newly deploy and delete Azure VM and Model server at the timing whenever we want using scheduler such as 'cron'.

Prerequisites

• Docker installed

Setup enviroment

docker pull mcr.microsoft.com/azure-cli:latest
docker run -it mcr.microsoft.com/azure-cli:latest

Continue inside the container from this command.

git clone https://github.com/hiouchiy/intel_ai_deploy_ovms_on_azure_vm.git
cd intel_ai_deploy_ovms_on_azure_vm

Setup a configuration file (JSON format)

Multiple model servers on single VM (deploy_config_singlevm.json)

You can deploy multiple model servers on single VM with configuration below. This configuraiton can save cost due to least number of VM but is basically better for test or develop use.

[
    {
        "vm_name": "modelservervm",
        "vm_size": "Standard_D2s_v4",
        "deploy_type": "vmss",
        "models":[
            {
                "function_name": "humanpose",
                "model_name": "human-pose-estimation",
                "env_name_ip": "HUMANPOSE_IPADDRESS",
                "env_name_port": "HUMAN_POSE_PORT",
                "port_number": 9000,
                "model_path_on_azure_storage": "az://ovms/intel/human-pose-estimation-0001/FP16-INT8",
                "model_server_version": "latest"
            },
            {
                "function_name": "handwritten",
                "model_name": "handwritten-japanese-recognition",
                "env_name_ip": "HANDWRITTEN_IPADDRESS",
                "env_name_port": "HAND_WRITTEN_PORT",
                "port_number": 9001,
                "model_path_on_azure_storage": "az://ovms/intel/handwritten-japanese-recognition-0001/FP16-INT8",
                "model_server_version": "latest"
            },
            {
                "function_name": "colorization",
                "model_name": "colorization",
                "env_name_ip": "COLORIZATION_IPADDRESS",
                "env_name_port": "COLORIZATION_PORT",
                "port_number": 9002,
                "model_path_on_azure_storage": "az://ovms/public/colorization-v2/FP32",
                "model_server_version": "2021.1"
            }
        ]
    }
]

Single model server on single VM (deploy_config.json)

You can deploy single model server on single VM with configuration below. In short, you need same number of vm as the number of models. This configuration is much more for production use than previous one.

[
    {
        "vm_name": "humanpose_vm",
        "vm_size": "Standard_D2s_v4",
        "deploy_type": "vmss",
        "models": [
            {
                "function_name": "humanpose",
                "model_name": "human-pose-estimation",
                "env_name_ip": "HUMANPOSE_IPADDRESS",
                "env_name_port": "HUMAN_POSE_PORT",
                "port_number": 9000,
                "model_path_on_azure_storage": "az://ovms/intel/human-pose-estimation-0001/FP16-INT8",
                "model_server_version": "latest"
            }
        ]
    },
    {
        "vm_name": "handwritten_vm",
        "vm_size": "Standard_D2s_v4",
        "deploy_type": "vmss",
        "models": [
            {
                "function_name": "handwritten",
                "model_name": "handwritten-japanese-recognition",
                "env_name_ip": "HANDWRITTEN_IPADDRESS",
                "env_name_port": "HAND_WRITTEN_PORT",
                "port_number": 9000,
                "model_path_on_azure_storage": "az://ovms/intel/handwritten-japanese-recognition-0001/FP16-INT8",
                "model_server_version": "latest"
            }
        ]
    },
    {
        "vm_name": "colorization_vm",
        "vm_size": "Standard_D2s_v4",
        "deploy_type": "vmss",
        "models": [
            {
                "function_name": "colorization",
                "model_name": "colorization",
                "env_name_ip": "COLORIZATION_IPADDRESS",
                "env_name_port": "COLORIZATION_PORT",
                "port_number": 9000,
                "model_path_on_azure_storage": "az://ovms/public/colorization-v2/FP32",
                "model_server_version": "2021.1"
            }
        ]
    }
]

Parameter Definitions

1. vm_name: a name of Azure VM
2. vm_size: a size of Azure VM
3. deploy_type: vm (single vm) / vmss (VM ScaleSet)
4. models: a list consisting of model(s) to be deoloyed on the VM named "vm_name"
   1. function_name: the name of the corresponding fucntion on Azure functions
   2. model_name: a unique name of deployed model on model server
   3. env_name_ip: Name of environemnt variable for IP address on Azure Functions
   4. env_name_port: Name of environemnt variable for port number on Azure Functions
   5. port_number: specific port number for accesing the model
   6. model_path_on_azure_storage: The path of model file (IR) on Azure Blob Storage. The format is az://container_name/folder_name.

Start deployment

First, login to Azure.

az login

If you want to login without web browser, follow the instrunctions below to setup service principle. Refer this site for the details.

az ad sp create-for-rbac --name ${DisplayName} --create-cert

Below result will be output.

{
  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "displayName": "${DisplayName}",
  "fileWithCertAndPrivateKey": "/home/user/xxxxxxxxxxx.pem",
  "name": "http://${DisplayName}",
  "password": null,
  "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

After that, you can login with below command.

az login --service-principal \
         --username ${appId} \
         --tenant ${tenantId} \
         --password ${fileWithCertAndPrivateKey} \

Finally, run the command below.

• For multiple models on single VM

  source ./deploy_vm.sh deploy_config_singlevm.json ResourceGroupName  "AzureStorageConnectionString"

  (*) AzureStorageConnectionString should be enclosed in double-quote.

  For example

  source ./deploy_vm.sh deploy_config_singlevm.json OVaaS_VMRG  "DefaultEndpointsProtocol=https;AccountName=…"

• For single model on single VM

  source ./deploy_vm.sh deploy_config.json ResourceGroupName  "AzureStorageConnectionString"

  (*) AzureStorageConnectionString should be enclosed in double-quote.

  For example

  source ./deploy_vm.sh deploy_config.json OVaaS_VMRG  "DefaultEndpointsProtocol=https;AccountName=…"

Parameters

1. The path of the configuration file
2. Specific name of resource group to be created
3. Azure storage connection string (SAS is also be accepted)

Delete Resource Group

Use Azure CLI directly as below.

az group delete --name ResourceGroupName --y

How to deploy OpenVINO™ Model Server automatically onto Azure VM for OVaaS from Azure Functions

Refer here.