Skip to content

A Kubernetes Operator that redeploys an app when resource limits are exceeded.

License

Notifications You must be signed in to change notification settings

secat/memhog-operator

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Creating Custom Operators

The memhog-operator is an example on how to create custom operators for Kubernetes.

The purpose of the memhog-operator is to watch for Pods in a namespace and monitor its memory usage. If the memory consumption of the Pod crosses a threshold, it will be vertically autoscaled by the operator.

Specifically, the operator will deploy a new copy of the Pod with a higher set of resource requests and limit, and then terminate the original Pod. The details of the higher resources are held within an AppMonitor, a custom TPR.

memhog: An example Pod that this operator would monitor.

Note: The memhog-operator is strictly for demo purposes. It is not intended to be used for any other use-cases.

Operator Structure

The memhog-operator is a combination of a custom ThirdPartyResource (TPR) known as the AppMonitor, and a custom controller to enforce state.

The AppMonitor encapsulates the autoscaling details for a Pod. The controller watches a Namespace for an AppMonitor, and for Pods that wish to be monitored (via Annotation). It then applies the operational thresholds and requirements declared in the AppMonitor onto the Pod.

Process

  • To monitor the Pod's resource memory consumption, the operator requires that the Pod have an annotation in its spec.template.metadata to associate itself with the memhog-operator.

    e.g. The memhog example is annotated as such:

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: memhog
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            name: memhog
          annotations:
            app-monitor.kubedemo.com/monitor: "true"
        spec:
          containers:
          - name: memhog
            image: quay.io/metral/memhog:v0.0.1
            imagePullPolicy: Always
            resources:
              limits:
                memory: 384Mi
              requests:
                memory: 256Mi
            ...
    
  • The Pod runs with a default set of resource requests & limits.

  • In the Pod's namespace there must also be an instantiated object of the custom AppMonitor TPR that the operator depends on e.g.:

    apiVersion: kubedemo.com/v1
    kind: AppMonitor
    metadata:
      name: mymonitor
    spec:
      memThresholdPercent: 75   # Percentage of (memory used) / (memory limit)
      memMultiplier: 2          # Multiplier factor used to increase memory resource requests & limits
    
  • The memhog-operator will watch the Pod's memory usage by querying Prometheus, apply the AppMonitor to the Pod as memory usage is retrieved, and redeploy the Pod with higher resource requests & limits if the AppMonitor thresholds are crossed.

  • If the Pod is redeployed, it will have updated resource requests & limits e.g.:

  ...
  resources:
    limits:
      memory: 768Mi
    requests:
      memory: 512Mi
  ...

Building & Running

// Build
$ glide up -v
$ make

Note: Prometheus is assumed to be running on http://localhost:9090

// Create cluster role & cluster role binding to work with TPR's.
$ kubectl create -f k8s/roles/role.yaml

// Run the operator
$ $GOPATH/bin/memhog-operator -v2 --prometheus-addr=http://localhost:9090 --kubeconfig=$HOME/.kube/config

About

A Kubernetes Operator that redeploys an app when resource limits are exceeded.

Resources

License

Stars

Watchers

Forks

Packages

No packages published