Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce cardinality in Dapr metrics and add more information to API logs #6919

Merged
merged 14 commits into from
Oct 24, 2023

Commits on Oct 20, 2023

  1. Reduce cardinality in Dapr metrics and add more information to API logs

    Fixes dapr#6723
    
    Includes:
    
    1. Drastically reduce cardinality of metrics, especially those emitted by the HTTP server:
       - Also removes the possibility of PII being included in the metrics endpoint.
    1. Add more information to API logs for both HTTP and gRPC, including things that are not going to be included in metrics by default anymore:
       - Response status code
       - Response size (for HTTP only & if possible)
       - Latency
    3. When `obfuscateURLs` is enabled for API logging, in API logs we now include a more descriptive name for the method rather than the path that was matched in the router. The descriptive names map to the names of the methods in gRPC, for example `SaveState` in place of `POST /state/{storeName}/{name}`. Since gRPC API logs are always, only "obfuscated" (because the params are in the body and not in the "URL"), this makes HTTP and gRPC API logs more consistent too
    4. Refactors how tracing and API logging (and the API token auth middleware) get the data from the handler/router:
       - The new approach is a lot more declarative, and less based on heuristics (such as parsing the path from the URL again)
       - The new approach also reduces the number of maps that are allocated and used in each request, which generally contained duplicate information generated in multiple parts of the code
    5. For both HTTP and gRPC, the way metadata is added to a tracing span has changed and it's now more declarative:
       - When looking at how tracing spans were composed, the metadata was added in `pkg/diagnostics`, separately from the endpoints. Only a VERY SMALL number of APIs had proper tracing configured (as a matter of fact, see the number of "TODO"'s in this PR), in large part due to the fact that this was in a separate package. The approach also relied a lot on heuristics.
       - For HTTP, now the `Endpoint` struct contains a property to add a function that adds properties to a span for tracing purposes. This lives right next to the handler.
       - For gRPC, messages defined in the protos whose name ends in "Request" now must define an `AppendSpanAttribute` method (this is enforced by a unit test)
    6. Update API Allowlisting/Denylisting for HTTP to:
       - Make sure that the same constants can be used for both HTTP and gRPC, especially versions. Right now, versions are in the format "v1.0-alpha1" for HTTP, and "v1alpha1" for gRPC. This PR changes the HTTP versions to be in the format "v1alpha1" too ("v1.0-alpha1" is preserved for bacwkards-compatibility)
       - Improved perf of the HTTP Allowlist/Denylist, especially when using the "new" format (e.g. versions with "v1alpha1"): checking the allowlist is now a simple map lookup rather than an iteration over the entire allowlist.
    
    Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
    ItalyPaleAle committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    2012ffd View commit details
    Browse the repository at this point in the history
  2. Added missing workflow beta1 APIs

    Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
    ItalyPaleAle committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    fbb3624 View commit details
    Browse the repository at this point in the history
  3. Added IT for metrics

    Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
    ItalyPaleAle committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    e95974e View commit details
    Browse the repository at this point in the history
  4. Update tests/integration/framework/binary/binary.go

    Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
    ItalyPaleAle committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    80ab29a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7da569d View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2023

  1. Configuration menu
    Copy the full SHA
    d8761dd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3fb6b3d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4538ed2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    07d74da View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    30bfca0 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. Configuration menu
    Copy the full SHA
    62899a1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    50d6074 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Configuration menu
    Copy the full SHA
    2803b4b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4d31a47 View commit details
    Browse the repository at this point in the history