Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add the tracing layer to enhance rest api metrics #4392

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

etolbakov
Copy link
Contributor

@etolbakov etolbakov commented Jan 15, 2024

Description

add the tracing layer to enhance rest api metrics

How was this PR tested?

The proposed changes have been deployed locally and verified with the /metrics endpoint. Please find the screenshot below:

Screenshot 2024-01-16 at 21 24 18

Related issue

#4076

@etolbakov
Copy link
Contributor Author

agreed with @fmassot to adapt the axum-prometheus approach and add gauge for pending requests

@etolbakov
Copy link
Contributor Author

@fmassot could you please take a look when you have time?

_latency: std::time::Duration,
_span: &tracing::Span,
) {
let labels = self.labels.lock().expect("Failed to unlock labels").clone();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to extract method_and_path obtaining in a method but it didn't compile. However, the gut feeling says that it should be possible

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not checked, but I suspect the span stores the method and path, and hopefully, we can access those fields later to populate the labels.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this point!
I've checked by simply putting dbg!(_span); in both on_failure and on_response and seems that the structure is barely informative:

[quickwit-serve/src/metrics.rs:135] _span = Span {
    name: "request",
    level: Level(
        Debug,
    ),
    target: "tower_http::trace::make_span",
    disabled: true,
    module_path: "tower_http::trace::make_span",
    line: 109,
    file: "/Users/etolbakov/.cargo/index.crates.io-6f17d22bba15001f/tower-http-0.4.4/src/trace/make_span.rs",
}

quickwit/quickwit-serve/src/metrics.rs Outdated Show resolved Hide resolved
_latency: std::time::Duration,
_span: &tracing::Span,
) {
let labels = self.labels.lock().expect("Failed to unlock labels").clone();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not checked, but I suspect the span stores the method and path, and hopefully, we can access those fields later to populate the labels.

quickwit/quickwit-serve/src/metrics.rs Outdated Show resolved Hide resolved
@etolbakov
Copy link
Contributor Author

@fmassot @guilload
Could you please take another look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants