Fixes https://github.com/opendatahub-io/model-registry/issues/174
Fixes https://github.com/opendatahub-io/model-registry/issues/175

Description

Implement the Model Registry <--> Model Serving interaction described in issue https://github.com/opendatahub-io/model-registry/issues/104.

Setup a new Reconciler that is triggered whenever a ServingRuntime is create/updated/deleted and in a periodic fashion (i.e., every N seconds where N could be specified as startup flag, default 60 seconds).

The summarized reconciliation workflow is the following:

• For each <Namespace>/<ServingRuntime> do:
  • Fetch all Inference services stored in the model registry and associated to the current NS (modeled as ServingEnvironment) and having as Runtime the current ServingRuntime name.
  • For each model registry <InferenceService> do:
    • Check if an InferenceService CR already exists for the current IS (use a custom label to keep track of these associations)
    • Check the model registry IS state
    • Then:
      • if STATE==DEPLOYED and not existing in cluster --> Create new ISVC
      • if STATE==UNDEPLOYED and not existing in cluster --> Do nothing, skip as there is no intention to deploy and there is no deployment in the cluster for that IS
      • if STATE==DEPLOYED and existing in cluster --> Check if there is any delta, if so apply updated CR
      • if STATE==UNDEPLOYED and existing in cluster --> This is an intention to undeploy the model, therefore delete the ISVC

ODH model controller keep track of those operation in the model registry itself, specifically as ServeModel recods (one per deployment).
These records can have different states, for now I used the following:

• RUNNING: deployed model (associated to the ISVC) is running in the namespace
• FAILED: Something went wrong creating/updating/deleting the ISVC - more details can be found in the CR itself
• UNKNOWN: Temporary and intermediate steps when the actual state of the operation is not known yet
• COMPLETE: The ISVC has been correctly removed, the deployment is removed - completed succesfully and not running anymore
• NEW: Not used for now

How Has This Been Tested?

Setup cluster

1. Install ODH 2.4.0
2. Install ODH component using the following DataScienceCluster CRD

kind: DataScienceCluster
apiVersion: datasciencecluster.opendatahub.io/v1
metadata:
  name: default
  labels:
    app.kubernetes.io/name: datasciencecluster
    app.kubernetes.io/instance: default
    app.kubernetes.io/part-of: opendatahub-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: opendatahub-operator
spec:
  components:
    codeflare:
      managementState: Removed
    dashboard:
      managementState: Managed
    datasciencepipelines:
      managementState: Managed
    kserve:
      managementState: Managed
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: overlays/odh
            uri: https://github.com/opendatahub-io/kserve/tarball/master
          - contextDir: config
            uri: https://github.com/lampajr/odh-model-controller/tarball/mr_serving_reconciler_test
    modelmeshserving:
      managementState: Managed
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: overlays/odh
            uri: https://github.com/opendatahub-io/modelmesh-serving/tarball/main
          - contextDir: config
            uri: https://github.com/lampajr/odh-model-controller/tarball/mr_serving_reconciler_test
    ray:
      managementState: Removed
    workbenches:
      managementState: Managed
    trustyai:
      managementState: Removed

NOTE: mr_serving_reconciler_test contains manifest changes to make use of the correct odh-model-controller image: alampare/odh-model-controller

By default this new model-registry for model-serving reconciler/controller is disabled in odh-model-controller, in order to properly enable it you should just start the controller with --model-registry-enabled flag.
https://github.com/lampajr/odh-model-controller/tarball/mr_serving_reconciler_test already contains this change in the configuration.

3. Install model registry operator
4. Clone model-registry-operator repository: git clone https://github.com/opendatahub-io/model-registry-operator.git
5. Install CRDs: make KUBECTL=oc install
6. Deploy the operator: make KUBECTL=oc deploy

Setup DS project

1. Create a Data Science project using the ODH dashboard, e.g., demo-model-registry-20231211
2. Setup a data connection (you could install a local minio instance in the newly create DS project)
3. Create a model server using the ODH dashboard
4. Apply the sample model registry CR from the model-registry-operator folder: kustomize build config/samples | oc apply -f - (make sure oc is pointing to the data science project create in step 1.)
5. Create a new model server using the ODH dashboard, in my case named model-server

Upload models

For the sake of simplicity let's upload some onnx files in the local minio instance (bucket name is models):

• models/mnist/v12/mnist-12.onnx
• models/mnist/v8/mnist-8.onnx

Fill up the model registry

NOTE: IDs could change depending on the order of operations and on the number of models you are going to register.

0. Set the model registry proxy url, you might have to setup a custom manual route from port :8080 of the model registry proxy

MR_HOSTNAME="<endpoint>"

1. Register an empty model, e.g, MNIST

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/registered_models" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "MNIST model for recognizing handwritten digits",
  "name": "MNIST"
}'

2. Add a new model version

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "MNIST model version n. 8",
  "name": "v8",
  "registeredModelID": "2"
}'

3. Add the model artifact to the created model version

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions/3/artifacts" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "Model artifact for MNIST v8",
  "uri": "s3://models/mnist/v8/mnist-8.onnx",
  "state": "UNKNOWN",
  "name": "v8-model",
  "modelFormatName": "onnx",
  "modelFormatVersion": "1",
  "storageKey": "aws-connection-models",
  "storagePath": "mnist/v8",
  "artifactType": "model-artifact"
}'

4. [Optional] If for some reason you need to update the artifact you can run:

curl --silent -X 'PATCH' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_artifacts/1" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '  {
    "description": "Model artifact for MNIST v8",
    "modelFormatName": "onnx",
    "modelFormatVersion": "1",
    "state": "UNKNOWN",
    "storageKey": "aws-connection-models",
    "storagePath": "mnist/v8",
    "uri": "s3://models/mnist/v8/mnist-8.onnx"
  }
'

NOTE: modelFormatName, modelFormatVersion, storageKey and storagePath are all those information needed to create the InferenceService so they should be valid values. E.g., the storageKey must match the existing data connection name (which is aws-connection-{{connection name}}).

You can use the following curls to inspect model registry content:

# Get registered models
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/registered_models?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

# Get model versions for a specific registered model
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

# Get model artifacts for a specific model version
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions/3/artifacts?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

# Get all inference services stored in model registry
curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

Test model controller workflow

As soon as you create a model server in the DS project, the odh-model-controller will create the ServingEnvironment in the model registry having the name == namespace (i.e., DS project), you can inspect its ID:

curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/serving_environments?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

1. Set the intention to deploy a model, i.e., create the inference service in the model registry:

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/serving_environments/1/inference_services" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "mnist-deployment",
  "runtime": "model-server",
  "registeredModelId": "2",
  "servingEnvironmentId": "1",
  "state": "DEPLOYED"
}'

The runtime must match the name of the model server create in the ODH platform for the current DS project.

In this specific example we are going to store in the model registry the intention to deploy a model into model-server Openvino server:

• servingEnvironmentId identifies the namespace where the deployment should occur, in this case id 1 which identifies DS project demo-model-registry-20231211
• registeredModelId identifies the registered model we would like to deploy
• modelVersionId identifies the specific version to deploy, if omitted (as in this case) let's deploy the latest version for that registered model

NOTE: the IDs must match, therefore ensure you are providing the corrects IDs.

At this point, the odh-model-controller will periodically inspect all existing ServingRuntime CRs (identifying configured model servers) and for each of them it will retrieve all Inference Services stored in model registry (using ServingEnv and Runtime as search criteria), then for each IS in model registry will do:

• if STATE==DEPLOYED and not existing in cluster --> Create new ISVC
• if STATE==UNDEPLOYED and not existing in cluster --> Do nothing, skip as there is no intention to deploy and there is no deployment in the cluster for that IS
• if STATE==DEPLOYED and existing in cluster --> Check if there is any delta, if so apply updated CR
• if STATE==UNDEPLOYED and existing in cluster --> This is an intention to undeploy the model, therefore delete the ISVC

❯ curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'
[
  {
    "createTimeSinceEpoch": "1702298833638",
    "customProperties": {},
    "id": "4",
    "lastUpdateTimeSinceEpoch": "1702298833638",
    "name": "mnist-deployment",
    "registeredModelId": "2",
    "runtime": "model-server",
    "servingEnvironmentId": "1",
    "state": "DEPLOYED"
  }
]

2. Tag a new version

Let's try to tag a new version and see if the reconciliation will automatically update the CR, triggering the atuomated deployment of the latest version (the new one)

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "MNIST model version n. 12",
  "name": "v12",
  "registeredModelID": "2"
}'

curl --silent -X 'POST' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/model_versions/5/artifacts" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "description": "Model artifact for MNIST v12",
  "uri": "s3://models/mnist/v12/mnist-12.onnx",
  "state": "UNKNOWN",
  "name": "v12-model",
  "modelFormatName": "onnx",
  "modelFormatVersion": "1",
  "storageKey": "aws-connection-models",
  "storagePath": "mnist/v12",
  "artifactType": "model-artifact"
}'

The next reconciliation loop should find that there is a Delta since the latest version is different, therefore the desired CR does not match the existing one. Once th eCR is updated, the model server should automatically react to deploy the new model:

1.702304193389716e+09 INFO OpenVINO Adapter.Openvino Model Server Adapter Server Both bucket and default_bucket params were provided in S3 storage config, ignoring default_bucket {"bucket": "models", "default_bucket": "models"}
1.7023041933897424e+09 DEBUG OpenVINO Adapter.Openvino Model Server Adapter Server found existing client in cache {"type": "s3", "cacheKey": "s3|0xc5ece145e9ad40f"}
1.7023041933943875e+09 DEBUG OpenVINO Adapter.Openvino Model Server Adapter Server found objects to download {"type": "s3", "cacheKey": "s3|0xc5ece145e9ad40f", "path": "mnist/v12", "count": 1}
1.7023041933945765e+09 DEBUG OpenVINO Adapter.Openvino Model Server Adapter Server downloading object {"type": "s3", "cacheKey": "s3|0xc5ece145e9ad40f", "path": "mnist/v12/mnist-12.onnx", "filename": "/models/mnist-deployment__isvc-8755919844/v12/mnist-12.onnx"}
1.702304193396579e+09 INFO OpenVINO Adapter.Openvino Model Server Adapter Server Calculated disk size {"modelFullPath": "/models/mnist-deployment__isvc-8755919844/v12", "disk_size": 26143}

3. Set the intention to undeploy the model by setting the state to UNDEPLOYED:

curl --silent -X 'PATCH' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services/4" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "runtime": "model-server",
  "state": "UNDEPLOYED"
}'

Form the odh-model-controller logs:

2023-12-11T14:21:31Z	INFO	controllers.ModelRegistry	ISVC existing and IS state is UNDEPLOYED, removing ISVC: mnist-deployment	{"ServingRuntime": "model-server", "namespace": "demo-model-registry-20231211", "InferenceService": "mnist-deployment"}
2023-12-11T14:21:32Z	DEBUG	controllers.ModelRegistry	Delta found	{"ServingRuntime": "model-server", "namespace": "demo-model-registry-20231211", "InferenceService": "mnist-deployment", "delete": "mnist-deployment"}
2023-12-11T14:21:32Z	INFO	controllers.ModelRegistry	Deleted ISVC: mnist-deployment	{"ServingRuntime": "model-server", "namespace": "demo-model-registry-20231211", "InferenceService": "mnist-deployment"}

As a result the model deployment should disappear from the ODH dashboard too.

4. Deploy a specific version rather than the latest one, i.e., v8

curl --silent -X 'PATCH' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services/4" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "runtime": "model-server",
  "modelVersionId": "3",
  "state": "DEPLOYED"
}'

Here some logs from ovms-adapter:

1.7023048054436572e+09 DEBUG OpenVINO Adapter.Openvino Model Server Adapter Server creating new repository client {"type": "s3", "cacheKey": "s3|0xc5ece145e9ad40f"}
1.7023048054478123e+09 DEBUG OpenVINO Adapter.Openvino Model Server Adapter Server found objects to download {"type": "s3", "cacheKey": "s3|0xc5ece145e9ad40f", "path": "mnist/v8", "count": 1}
1.7023048054479847e+09 DEBUG OpenVINO Adapter.Openvino Model Server Adapter Server downloading object {"type": "s3", "cacheKey": "s3|0xc5ece145e9ad40f", "path": "mnist/v8/mnist-8.onnx", "filename": "/models/mnist-deployment__isvc-36ecdf0658/v8/mnist-8.onnx"}

5. The state and operations performed by odh-model-controller are audited in the model registry as ServeModel records (associated to a specific model registry inference service) that you can inspect as follows:

curl --silent -X 'GET' \
  "$MR_HOSTNAME/api/model_registry/v1alpha1/inference_services/4/serves?pageSize=100&orderBy=ID&sortOrder=DESC&nextPageToken=" \
  -H 'accept: application/json' | jq '.items'

[
  {
    "createTimeSinceEpoch": "1702304792636",
    "customProperties": {},
    "id": "2",
    "lastKnownState": "RUNNING",
    "lastUpdateTimeSinceEpoch": "1702304852848",
    "modelVersionId": "3",
    "name": "mnist-deployment/68cf3949-b9d6-4510-b5e6-4994ffdcd873"
  },
  {
    "createTimeSinceEpoch": "1702298852648",
    "customProperties": {},
    "id": "1",
    "lastKnownState": "COMPLETE",
    "lastUpdateTimeSinceEpoch": "1702304492637",
    "modelVersionId": "3",
    "name": "mnist-deployment/86cdc3d7-86e3-4ef9-a7c0-bbd5df7ba460"
  }
]

NOTE step Upload models and Fill up the model registry could be automated using either notebooks or pipelines, please follow a more complete e2e demo at https://github.com/tarilabs/demo20231121#demo

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work