Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mimir Alertmanager crashes when visiting the UI and ms teams receiver configured #8001

Open
jmichalek132 opened this issue Apr 29, 2024 · 1 comment
Open

Mimir Alertmanager crashes when visiting the UI and ms teams receiver configured #8001

jmichalek132 opened this issue Apr 29, 2024 · 1 comment
Labels
bug Something isn't working component/alertmanager

Comments

@jmichalek132
Copy link
Contributor

Describe the bug

Visiting the Mimir alertmanager UI, for a tenant that uses the fallbackConfig when configured with msteams receiver causes alertmanager to crash. Removing the msteams fixes it.

To Reproduce

Steps to reproduce the behavior:

  1. Deploy mimir helm chart into Kubernetes cluster using the mentioned helm chart and version
  2. Enable Mimir alertmanager with configuration (using the fallbackConfig option in the helm chart)
      receivers:
      - name: 'another'
        webhook_configs:
        - send_resolved: true
          url: 'http://xxxxxx:80/alert' 
      - name: "null"
      - name: "msteams"
        msteams_configs:
          - send_resolved: true
            webhook_url: "https://xxxxxxxxxxxxxxxxxx"
            http_config:
              proxy_url: "http://xxxxxxxxxxxxxxxxxx:8080/"
              tls_config:
                ca_file: "/etc/ssl/certs/xxxxxxxxxx.crt"
      route:
        group_by: ['alertname', 'service']
        group_wait: 30s
        group_interval: 1m
        repeat_interval: 1m
        receiver: 'another'
        routes:
        - receiver: 'null'
          matchers:
            - alertname =~ "InfoInhibitor|Watchdog"
        - receiver: 'msteams'
          matchers:
            - ci =~ "test.dev"
  1. Visit Alertmanager ui for a tenant without it's own alertmanager config (falling back on the fallbackConfig) or reproduce an api call done by the frontend using curl
curl 'http://localhost:8080/alertmanager/api/v2/alerts/groups?silenced=false&inhibited=false&active=true' \
  -H 'Accept: */*' \
  -H 'Accept-Language: en-US,en;q=0.9,sk-SK;q=0.8,sk;q=0.7,es-ES;q=0.6,es;q=0.5' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'DNT: 1' \
  -H 'Pragma: no-cache' \
  -H 'Referer: http://localhost:8080/alertmanager/' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'x-scope-orgid: metamonitoring'
  1. Mimir alertmanager crashes

Expected behavior

Mimir alertmanager to not crash when web UI visited, given MS teams was introduced as supported integration in 2.10.0.

Environment

  • Infrastructure: AKS v1.28.3
  • Deployment tool: mimir-distributed helm chart version 5.3.0 mimir version 2.12.0

Additional Context

Logs from the mimir alertmanager instance.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x1160c4a]

goroutine 120 [running]:
github.com/prometheus/alertmanager/dispatch.(*Dispatcher).Groups(0x0, 0xc00453e720, 0xc004081620)
        /__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/dispatch/dispatch.go:222 +0x6a
github.com/grafana/mimir/pkg/alertmanager.New.func3(0x0?, 0x0?)
        /__w/mimir/mimir/pkg/alertmanager/alertmanager.go:271 +0x25
github.com/prometheus/alertmanager/api/v2.(*API).getAlertGroupsHandler(0xc0026c38c0, {0xc0045f2c00, 0xc003b7c52e, {0x0, 0x0, 0x0}, 0xc003b7c52f, 0x0, 0xc003b7c550})
        /__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/api/v2/api.go:405 +0x32d
github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup.GetAlertGroupsHandlerFunc.Handle(0xc0032547c0?, {0xc0045f2c00, 0xc003b7c52e, {0x0, 0x0, 0x0}, 0xc003b7c52f, 0x0, 0xc003b7c550})
        /__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup/get_alert_groups.go:33 +0x6b
github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup.(*GetAlertGroups).ServeHTTP(0xc002a3a990, {0x31e1a60, 0xc003f7c9c0}, 0xc0045f2c00)
        /__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup/get_alert_groups.go:67 +0x238
github.com/go-openapi/runtime/middleware.(*Context).RoutesHandler.NewOperationExecutor.func1({0x31e1a60, 0xc003f7c9c0}, 0xc0045f2c00)
        /__w/mimir/mimir/vendor/github.com/go-openapi/runtime/middleware/operation.go:28 +0x53
net/http.HandlerFunc.ServeHTTP(0x0?, {0x31e1a60?, 0xc003f7c9c0?}, 0x4ccb82?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/go-openapi/runtime/middleware.NewRouter.func1({0x31e1a60, 0xc003f7c9c0}, 0xc0045f2a00)
        /__w/mimir/mimir/vendor/github.com/go-openapi/runtime/middleware/router.go:80 +0x257
net/http.HandlerFunc.ServeHTTP(0xc0016dc0ad?, {0x31e1a60?, 0xc003f7c9c0?}, 0x500000000000010?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/go-openapi/runtime/middleware.Spec.func1({0x31e1a60, 0xc003f7c9c0}, 0xc0045f2a00)
        /__w/mimir/mimir/vendor/github.com/go-openapi/runtime/middleware/spec.go:72 +0x1b1
net/http.HandlerFunc.ServeHTTP(0xc001902900?, {0x31e1a60?, 0xc003f7c9c0?}, 0x29a533f?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/prometheus/alertmanager/api/v2.NewAPI.setResponseHeaders.func2({0x31e1a60, 0xc003f7c9c0}, 0x4105e5?)
        /__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/api/v2/api.go:149 +0x11e
net/http.HandlerFunc.ServeHTTP(0xc00315c7e0?, {0x31e1a60?, 0xc003f7c9c0?}, 0xc0045f2a00?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/rs/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x31e1a60, 0xc003f7c9c0}, 0xc0045f2a00)
        /__w/mimir/mimir/vendor/github.com/rs/cors/cors.go:281 +0x184
net/http.HandlerFunc.ServeHTTP(0x0?, {0x31e1a60?, 0xc003f7c9c0?}, 0xd?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/prometheus/alertmanager/api.(*API).Register.StripPrefix.func1({0x31e1a60, 0xc003f7c9c0}, 0xc0045f2900)
        /usr/local/go/src/net/http/server.go:2179 +0x262
net/http.HandlerFunc.ServeHTTP(0xc003255fc0?, {0x31e1a60?, 0xc003f7c9c0?}, 0xc001902b28?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/prometheus/alertmanager/api.(*API).limitHandler.func1({0x31e1a60?, 0xc003f7c9c0?}, 0xc0045f2900?)
        /__w/mimir/mimir/vendor/github.com/prometheus/alertmanager/api/api.go:211 +0x1d2
net/http.HandlerFunc.ServeHTTP(0x1400000000000001?, {0x31e1a60?, 0xc003f7c9c0?}, 0xc00453e660?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
net/http.(*ServeMux).ServeHTTP(0x2502780?, {0x31e1a60, 0xc003f7c9c0}, 0xc0045f2900)
        /usr/local/go/src/net/http/server.go:2514 +0x142
github.com/grafana/mimir/pkg/alertmanager.(*MultitenantAlertmanager).serveRequest(0xc001b602c0, {0x31e1a60, 0xc003f7c9c0}, 0xc0045f2900)
        /__w/mimir/mimir/pkg/alertmanager/multitenant.go:873 +0x6f5
github.com/grafana/mimir/pkg/alertmanager.(*handlerForGRPCServer).ServeHTTP(0x31ea220?, {0x31e1a60?, 0xc003f7c9c0?}, 0x418628?)
        /__w/mimir/mimir/pkg/alertmanager/multitenant.go:439 +0x25
github.com/grafana/dskit/httpgrpc/server.Server.Handle({{0x31cb840?, 0xc001b102d0?}, 0xfa?}, {0x31ea220?, 0xc0040814a0?}, 0x27a4900?)
        /__w/mimir/mimir/vendor/github.com/grafana/dskit/httpgrpc/server/server.go:68 +0x115
github.com/grafana/mimir/pkg/alertmanager.(*MultitenantAlertmanager).HandleRequest(0x7f2460233a68?, {0x31ea220?, 0xc0040814a0?}, 0x25bed40?)
        /__w/mimir/mimir/pkg/alertmanager/multitenant.go:858 +0x3a
github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb._Alertmanager_HandleRequest_Handler.func1({0x31ea220, 0xc0040814a0}, {0x28cc3c0?, 0xc003b40870})
        /__w/mimir/mimir/pkg/alertmanager/alertmanagerpb/alertmanager.pb.go:472 +0x72
github.com/grafana/mimir/pkg/mimir.ThanosTracerUnaryInterceptor({0x31ea220?, 0xc004081470?}, {0x28cc3c0, 0xc003b40870}, 0x31ea220?, 0xc003ea2390)
        /__w/mimir/mimir/pkg/mimir/tracing.go:26 +0x69
google.golang.org/grpc.getChainUnaryHandler.func1({0x31ea220, 0xc004081470}, {0x28cc3c0, 0xc003b40870})
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1205 +0xb2
github.com/grafana/dskit/middleware.ServerUserHeaderInterceptor({0x31ea220?, 0xc0040813e0?}, {0x28cc3c0, 0xc003b40870}, 0xc0046fedc0?, 0xc003f7c980)
        /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_auth.go:43 +0x5b
github.com/grafana/mimir/pkg/util/noauth.SetupAuthMiddleware.func1({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870}, 0xc0033604e0, 0xc003f7c980)
        /__w/mimir/mimir/pkg/util/noauth/no_auth.go:32 +0x9d
google.golang.org/grpc.getChainUnaryHandler.func1({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870})
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1205 +0xb2
github.com/grafana/mimir/pkg/querier/api.ReadConsistencyServerUnaryInterceptor({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870}, 0xc0033604e0?, 0xc003f7c940)
        /__w/mimir/mimir/pkg/querier/api/consistency.go:78 +0x126
google.golang.org/grpc.getChainUnaryHandler.func1({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870})
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1205 +0xb2
github.com/grafana/dskit/middleware.UnaryServerInstrumentInterceptor.func1({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870}, 0xc0033604e0, 0xc003f7c900)
        /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:33 +0xa9
google.golang.org/grpc.getChainUnaryHandler.func1({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870})
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1205 +0xb2
github.com/grafana/dskit/server.newServer.HTTPGRPCTracingInterceptor.func2({0x31ea220?, 0xc0040813e0?}, {0x28cc3c0?, 0xc003b40870?}, 0xc0033604e0?, 0xc003ea2390?)
        /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/http_tracing.go:75 +0x947
google.golang.org/grpc.getChainUnaryHandler.func1({0x31ea220, 0xc0040813e0}, {0x28cc3c0, 0xc003b40870})
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1205 +0xb2
github.com/opentracing-contrib/go-grpc.OpenTracingServerInterceptor.func1({0x31ea220, 0xc004080ff0}, {0x28cc3c0, 0xc003b40870}, 0xc0033604e0, 0xc003f7c740)
        /__w/mimir/mimir/vendor/github.com/opentracing-contrib/go-grpc/server.go:57 +0x3ea
google.golang.org/grpc.getChainUnaryHandler.func1({0x31ea220, 0xc004080ff0}, {0x28cc3c0, 0xc003b40870})
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1205 +0xb2
github.com/grafana/dskit/middleware.GRPCServerLog.UnaryServerInterceptor({{0x31c9e60?, 0xc0006bd368?}, 0x0?, 0x6c?}, {0x31ea220, 0xc004080ff0}, {0x28cc3c0?, 0xc003b40870?}, 0xc0033604e0, 0xc003f7c700)
        /__w/mimir/mimir/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:49 +0xc4
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x31ea220, 0xc004080ff0}, {0x28cc3c0, 0xc003b40870}, 0xc00198d960?, 0x2538fe0?)
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1196 +0x85
github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb._Alertmanager_HandleRequest_Handler({0x293d980?, 0xc001b602c0}, {0x31ea220, 0xc004080ff0}, 0xc00377c200, 0xc000794220)
        /__w/mimir/mimir/pkg/alertmanager/alertmanagerpb/alertmanager.pb.go:474 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00167f600, {0x31ea220, 0xc004080ed0}, {0x3206320, 0xc003da01a0}, 0xc003e12ea0, 0xc001b80c30, 0x47343c0, 0x0)
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1385 +0xe03
google.golang.org/grpc.(*Server).handleStream(0xc00167f600, {0x3206320, 0xc003da01a0}, 0xc003e12ea0)
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1796 +0xfec
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:1029 +0x8b
google.golang.org/grpc.(*Server).serverWorker(0xc00167f600)
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:632 +0x25
created by google.golang.org/grpc.(*Server).initServerWorkers in goroutine 1
        /__w/mimir/mimir/vendor/google.golang.org/grpc/server.go:645 +0x112
@56quarters 56quarters added bug Something isn't working component/alertmanager labels Apr 29, 2024
@jmichalek132
Copy link
Contributor Author

I tried to reproduce it with upstream alertmanager locally, with the same alertmanager config but couldn't. The version of upstream manager tested with:

alertmanager, version 0.27.0 (branch: HEAD, revision: 0aa3c2aad14cff039931923ab16b26b7481783b5)
  build user:       root@22cd11f671e9
  build date:       20240228-11:51:20
  go version:       go1.21.7
  platform:         linux/amd64
  tags:             netgo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component/alertmanager
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@56quarters @jmichalek132