Export OpenTelemetry traces during deploy

[sandydoo]

Resolves #562.

Haven't tested this yet.

[domenkozar]

@zowoq could you give this a try?

[zowoq]

Happy to test this but I'm not familiar with opentelemetry so I'll need to figure out how to wire it up to telegraf, seems to require more than just enabling the telegraf opentelemetry plugin.

[domenkozar]

Something like https://github.com/influxdata/telegraf/tree/master/plugins/inputs/opentelemetry

[zowoq]

Something like https://github.com/influxdata/telegraf/tree/master/plugins/inputs/opentelemetry

Does it require any specific config? As I said, just enabling the plugin doesn't seem to work?

[sandydoo]

The telegraf plugin only supports gRPC and our Haskell library only exports binary payloads via HTTP. 🤦

One way around this would be to run an instance of the OpenTelemetry Collector. It speaks both protocols.

[sandydoo]

You can use environment variables to change where and how the cachix client exports traces: https://opentelemetry.io/docs/concepts/sdk-configuration/otlp-exporter-configuration/. The default HTTP endpoint is http://localhost:4318.

AFAIK, it's recommended to run the collector locally on each machine and configure it to export to your remote datastore.

[zowoq]

My interest in this was that it would integrate into our existing monitoring, I am not going to run another service (either a central instance or locally on each machine) just to pull metrics from a single service.

Thank you anyway, I appreciate the effort that went into implementing this PR.

[sandydoo]

@zowoq, yeah, that makes sense. I wonder how much work it would be to get our end speak gRPC 🤔

[domenkozar]

My interest in this was that it would integrate into our existing monitoring, I am not going to run another service (either a central instance or locally on each machine) just to pull metrics from a single service.

Thank you anyway, I appreciate the effort that went into implementing this PR.

Note that you would only need to run telemetry-collector daemon on the cachix deploy nodes that would then push to your telegraf instance.

[zowoq]

Note that you would only need to run telemetry-collector daemon on the cachix deploy nodes that would then push to your telegraf instance.

Telegraf is a collector that we run on each machine that is scraped by our prometheus instance.