Telemetry Beta API

[a-thaler]

Description
The telemetry API should head towards a proper v1 version and requires the step to a beta version as next step. For the logs and trace API, smaller things should get adjusted like changing the status to be conform to k8s conventions (#601). For logs the Loki output needs to get removed and the API prepared for OTLP support (#556).

The goal is to get a streamlined API across all pipeline types with a betaV1 version, having:

• consistent status handling
• consistent OTLP output handling
• consistent input handling

Action:

• Preparations
  • Understand the k8s API restrictions and outline a path to a beta version (Path to a beta API version #712)
• Implementation
  • Align existing status API of logs and traces with syntactical kubernetes conventions (Align existing status API of logs and traces with syntactical kubernetes conventions #772)
  • Introduce new status API for logs and traces, mark the old one as deprecated (Introduce new status API for logs and traces, mark the old one as deprecated #805)
  • Prepare experimental beta feature: Introduce feature flag and adjust kustomize deployment to deploy dummy beta version only when active (Introduce beta API as experimental feature #848)
  • Introduce beta API for operator
  • Remove loki output from logs beta version ( Remove loki output from logs beta version and introduce initial conversion webhook for beta version #873)
  • Introduce conversion webhook for beta version ( Remove loki output from logs beta version and introduce initial conversion webhook for beta version #873)
  • Adjust input section in logs beta version (namespace selector, enabled flag, naming)
  • MIgrate parts of ValidatingWebhook of LogPipeline which require CRD changes
    1. CRD Validation
    - Move API validation into CRD schema validation
    - Move unsupported mode warning
  • Remove old stats from traces and logs in beta version
  • Remove LogParser LogParser removal #823
  • Remove feature flag
  • Make sure that Go structs are properly named (e.g. LogPipeline specific types should be prefixed with LogPipeline, which is currently not the case)
• Follow-Ups
  • Switch to namespace selector by labels (Determine system namespaces by label #380)
  • Remove ValidatingWebhook of LogPipeline
    2. Move validation into pipeline logic
    - Max Pipeline check to be executed on backend side similar to TracePipeline, having proper status change
    - Singleton check for telemetry resource [Telemetry Manager] Assure that only one module configuration is in use (CR singleton) #555
    - Plugin validation with supported list and hardcoded available plugin list as backend validation feature reflected via status change
    - Missing file/variable reference validation
    3. Cleanup
    - Delete dry-run and get manager image free
    - Delete and remove webhook

[k15r]

Important for the removal of the old crd version

• create a script to update all existing CRs to the latest version (read it from API-Server, write it back in the latest version)
• mark the old version as served: false, storage: false
• remove the old version from the Status Subressource of the CRD status.storedVersions
• remove the old version from the spec of the CR

keep track of this issue: kyma-project/lifecycle-manager#1545