Full test coverage for advanced status

[a-thaler]

Description

Before #425 can get finalized, the full feature must be covered by e2e tests. That mainly includes triggering every condition reason by a test setup.

Criterias

• All reasons of the new condition are tested

Actions

• Come up with a concept on how to test the feature. It is hard to test all situations. Maybe a contract test based on metrics is enough?
• Implement tests for logs
• Implement tests for traces
• Implement tests for metrics

Reasons

Attachments

Release Notes

[skhalash]

The following 2 test scenarios have been implemented for Log Pipelines:

• Backend outage (by downscaling the mock backend to 0 replicas): the pipeline healthiness gradually degrades to AllTelemetryDataDropped
• Backend backpressure (by deploying a Virtual Service with fault injection - 90% of requests get a 502 response): the pipeline healthiness gradually degrades to SomeTelemetryDataDropped

For both scenarios, there has been a separate job created. The jobs are stable and take <10 minutes to execute.

As the next step, we can implement the same scenarios for Metric and Trace Pipelines. What will not be covered is the gateway throttling scenario - I played around with it and didn't find a way to reproduce it even manually.

[skhalash]

In general, we want to test how OTel Collector and Fluent Bit instrumentation logic works. Contract based testing using mocks is not enough, we need the actual software running and negative scenarios simulated.