Enable consumption and configuration of specific hyperscaler resources

[varbanv]

Description

Provide a way for end users to consume and be charged for a pre-defined set of hyperscaler resources:

• specialized node types. For example GPU and ARM nodes, network, memory and CPU optimized nodes, etc.
• additional types of storage, including SSD and read-write-many options.
• Worker pool configuration for resources

Context

Problem

Currently, Kyma is a layer on top of Kubernetes and as such provides a very limited set of infrastructure configuration options at provisioning time.
However, customers looking to adopt Kyma that already use existing hyperscaler offerings already take advantage of specialized resources as part of their workloads (for example faster storage, GPU nodes, network optimized nodes, etc). This prevents those users from on-boarding on Kyma without having to re-engineer their workloads.

Benefits

For customers:

• greater flexibility around infrastructure requirements
• ability to meet requirements in order to move workloads to Kyma
• reduced maintenance related to hyperscaler accounts

For us:

• increase adoption
• abstract and bundle infrastructure related requirements in one feature
• mitigate the abandoned BYOC approach

Potential problems

• billing could become more complex if we don't introduce an as way for customers to track their costs

Acceptance criteria

• Users can enable/disable hyperscaler resource usage for their cluster from a pre-defined list of options
• Users can enable/disable hyperscaler resource usage via BTP Cockpit, Kyma Dashboard and command line
• Users will be billed for all additional usage in their clusters and the charges will be correctly reflected in their billing summary
• Users cannot select or enable hyperscaler resources that are not part of the pre-defined list.
• Kyma will provide a mechanism for end users to access relevant information related to the hyperscaler resource they use without providing direct access to the hyperscaler account
• List of available resources should be based on region and hyperscaler availability
• In case of quota/availability limitations for a specific resource, end users will be given reasonable information about why they cannot consume the desired resource

Tasks

• technical workshop to define the architecture
• decide which machine types/nodes to expose first
• node type input in cockpit
• selected node types passed to infrastructure manager
• KMC handles metering for different node type
• worker pool configuration input in cockpit - limited set of parameters exposed
• worker pool configuration passed to infrastructure manager
• provide separate/default worker pool for Kyma resources based on standard node types (non ARM, etc)
• limit Kyma resources to run on the dedicated Kyma worker pool
• KMC handles metering for non-basic storage types
• provide feedback to the end user about errors during resource configuration changes - e.g. no GPUs available from the hyperscaler
• document the pricing and calculation for the additional node types

[Disper]

• That would potentially require rewriting the provisioning/de-provisioning logic in the infrastructure manager, as we want to move away from the old provisioner.
• Question if the implementation would enable specific set of resources or a generic one, that we will with any types of resources? How will that pre-defined list look like?

[marco-porru]

cKMS team is evaluating the usage of Kyma.
The team need to have "confidential computing capabilities". This kind of machine is surely available for azure and gcp

[marco-porru]

SAP for Me would like to use m6g and m6in machine types

[valentinvieriu]

+1 for GPUs

[marco-porru]

+1 for GPUs

Thanks Valentin for reporting it. I think it's worth mentioning the context and let me do it on behalf of you for simplicity 😄 : it's to make it possible for Core AI to run on Kyma (subject to future discussions and agreements)

[varbanv]

Had a preliminary workshop with @tobiscr and @PK85 and added a first set of tasks to work on.

[marco-porru]

+1 team for GPU (ICN Munich)

[marco-porru]

Enable more private connectivity (e.g. via firewall), requested by not less than 3 teams (e.g. S/4HANA ABAP Machines)

[marco-porru]

Enable assured workload GCP module (relevant for KSA), requested by BTP email service

[abbi-gaurav]

A customer is looking for very high IOPS storage.
e.g. enabling ultra disks for storage could help them: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-enable-ultra-ssd?tabs=azure-portal

[abbi-gaurav]

At present, customers are able to use resources for which they are not charged such as

• additional load balancers
• non default storage class e.g. for multiple read write

We should somehow make the customers aware that they might have to pay for this in the future, so it should not come as a surprise for them.

@NHingerl , could you please help? IMHO, putting this info out might not need to wait until this epic is done.