Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable dedicated interceptor/scalers with the same operator #241

Open
Tracked by #911
arschles opened this issue Aug 25, 2021 Discussed in #218 · 4 comments · May be fixed by #1014
Open
Tracked by #911

Enable dedicated interceptor/scalers with the same operator #241

arschles opened this issue Aug 25, 2021 Discussed in #218 · 4 comments · May be fixed by #1014
Labels
feature stale-bot-ignore All issues that should not be automatically closed by our stale bot

Comments

@arschles
Copy link
Collaborator

arschles commented Aug 25, 2021

Discussed in #218

Originally posted by rwkarg August 4, 2021
This issue (#36) has been closed but I don't see a way to currently add annotations to the interceptor service.

The specific use case we have is using autoneg-controller to add workload instances from multiple clusters as backends to a single Load Balancer. In GCP, this requires that the service (would be the interceptor service) has two annotations to create the Network Endpoint Group (NEG) and to associate that NEG with a Backend Service on the Load Balancer (not important what it does, just that we need annotations on the interceptor service).

Looks like #206 may complicate this, if there is only one service. We can't add the annotations to individual services any more for autonet-controller to work in that case.

Is the multi-tenant interceptor something that can be opted out of?


Further along in that discussion, @tomkerkhove and I talked through how to allow operators to opt-out of the shared interceptor/scaler while still using the same operator. in this proposal, I outline how to enable that exact scheme.

New CRD: HTTPScalingComponents

The purpose of this new, optional, CRD, is to specify a set of running interceptor and scaler. Interceptor and scaler are always deployed together. When an HTTPScalingComponents CRD is submitted, the operator will react by ensuring the following are installed.

  • interceptor Deployment
  • interceptor Services for proxy and admin endpoints
  • interceptor ScaledObject
  • scaler Deployment
  • scaler Services for gRPC and health check/admin endpoints

Sample YAML for a HTTPScalingComponents CRD would be similar to:

kind: HTTPScalingComponents
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: MyApp
spec:
    interceptor:
        serviceType: ClusterIP # this is the default, but you can specify other service types
        minReplicas: 10
        maxReplicas: 100
        annotations:
            key1: val1
    scaler:
        serviceType: ClusterIP # same as spec.interceptor.serviceType
        annotations:
            key1: val1

Usage in HTTPScaledObjects

A user submitting an HTTPScaledObject will optionally be allowed to specify the components that they would like their app to use. The field, scalingComponents, would be used as follows:

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: MySO
spec:
    scalingInfrastructure:
        name: MyApp
    host: myhost.com
    scaleTargetRef:
        deployment: myappdeployment
        service: myappservice
        port: 8080
    replicas:
        min: 0
        max: 1000

Note the new spec.scalingInfrastructure section. spec.scalingInfrastructure.name specifies an already-installed HTTPScalingComponents CRD and is optional. The HTTP Addon operator is configured with a default HTTPScalingComponents on startup (and one is installed by default in the official helm chart). Users who omit this name field will simply use the default

Implementation notes

The biggest changes will come to the operator if this proposal is implemented. The changes of note are:

  • Create new scaler and interceptor infrastructure when a new HTTPScalingComponents is created
  • Segment routing tables by HTTPScalingComponents, and write one routing table ConfigMap per HTTPScalingComponents CRD
  • Tell the interceptor Deployment which aforementioned ConfigMap to consume
  • On HTTPScaledObject submission, create ScaledObjects that point to the proper scaler Service

cc/ @rwkarg @tomkerkhove

@arschles arschles changed the title Add annotations to interceptor service Use dedicated interceptor/scaler with the same operator Aug 25, 2021
@arschles arschles changed the title Use dedicated interceptor/scaler with the same operator Enable dedicated interceptor/scalers with the same operator Aug 25, 2021
@arschles
Copy link
Collaborator Author

Note that this functionality would interact with #240. See #240 (comment) for more details

@arschles arschles added this to the v0.3.0 milestone Oct 8, 2021
@arschles arschles modified the milestones: v0.3.0, v0.4.0 Nov 16, 2021
@stale
Copy link

stale bot commented Jan 15, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Jan 15, 2022
@arschles arschles added the stale-bot-ignore All issues that should not be automatically closed by our stale bot label Jan 20, 2022
@stale stale bot removed the stale All issues that are marked as stale due to inactivity label Jan 20, 2022
@arschles arschles added the stale All issues that are marked as stale due to inactivity label Jan 20, 2022
@stale
Copy link

stale bot commented Jan 27, 2022

This issue has been automatically closed due to inactivity.

@stale stale bot closed this as completed Jan 27, 2022
@arschles
Copy link
Collaborator Author

Reopening with label to keep the stale bot from closing this. This is still a valid feature that we'd like to explore

@arschles arschles reopened this Jan 28, 2022
@stale stale bot removed the stale All issues that are marked as stale due to inactivity label Jan 28, 2022
@tomkerkhove tomkerkhove removed this from the v0.5.0 milestone May 11, 2022
@JorTurFer JorTurFer mentioned this issue Apr 4, 2024
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature stale-bot-ignore All issues that should not be automatically closed by our stale bot
Projects
Status: To Do
Development

Successfully merging a pull request may close this issue.

2 participants