Observability planning

[agracey]

Discussion about what we can do to improve the observability of applications being built with Epinio.

[agracey]

Metrics:

• Resource consumption (and allowances) being passed back to the developer
• Maybe a way to see what metrics the app is exposing on /metrics?

Tracing:

• OpenTelemetry collector sidecar in application pods and configured to send to correct backend
• Easy way to deploy linkerd-jaeger (potentially to be renamed by them to linkerd-tracing or similar)
• Documentation on how to instrument code and use built-in collector sidecar

Logging:

• Structured logging collection if the libraries are ready. Which I'm not sure they are.
• Maybe some

Two notes:

• Some of this might be easily doable without code changes if there are projects that already do the sidecar injection
• While I don't expect this to be an issue, we should always fallback gracefully if an application doesn't instrument their code.