Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add e2e test on secret update for PrometheusAgent #6426

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jamshidi799
Copy link
Contributor

It fixes #6414

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

  • CHANGE (fix or feature that would cause existing functionality to not work as expected)
  • FEATURE (non-breaking change which adds functionality)
  • BUGFIX (non-breaking change which fixes an issue)
  • ENHANCEMENT (non-breaking change which improves existing functionality)
  • NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Verification

Please check the Prometheus-Operator testing guidelines for recommendations about automated tests.

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.


@jamshidi799 jamshidi799 requested a review from a team as a code owner March 23, 2024 20:24
test/e2e/prometheusagent_test.go Outdated Show resolved Hide resolved
test/e2e/prometheusagent_test.go Outdated Show resolved Hide resolved
test/e2e/prometheusagent_test.go Show resolved Hide resolved
@ArthurSens
Copy link
Member

Thanks for tackling this issue anyway!

I'm not entirely sure how this can be tested as well, some ideas that I had:

  • Measure the metrics exposed by the operator itself, I can see one metric that sounds useful:
# HELP prometheus_operator_triggered_total Number of times a Kubernetes object add, delete or update event triggered the Prometheus Operator to reconcile an object
# TYPE prometheus_operator_triggered_total counter
  • Access prometheus filesystem somehow and check the content of the mounted secret

@simonpasquier
Copy link
Contributor

Since we're looking at end-to-end tests, I would tackle it from the end-user's standpoint: "I updated the credentials used to connect to my target but the update was never propagated to Prometheus and scrapes started to fail".

In practice:

  • deploy a target which requires a valid bearer token
  • configure a monitor referring to a secret containing an invalid bearer token
  • wait for Prometheus to report the target as down
  • update the secret content with the good token
  • wait for Prometheus to report the target as up

Signed-off-by: Mohammad Jamshidi <jamshidi.m799@gmail.com>
Copy link
Member

@ArthurSens ArthurSens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm struggling a bit to understand the correctness of the test. Is abc a valid bearertoken? Could we add comments to the code explaining the test a little bit?

t.Fatal(err)
}

time.Sleep(time.Minute * 2)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe WaitForHealthyTargets already waits until it finds the healthy targets. What do you think about removing this sleep then?

Comment on lines +278 to +280
if err := framework.WaitForHealthyTargets(context.Background(), ns, "prometheus-agent-operated", 1); err == nil {
t.Fatal("All targets should be down")
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If all targets should be down, why do we look for 1 healthy target?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add e2e test on secret update for PrometheusAgent
3 participants