Application Debugging Tools

[agracey]

I wanted a place to brainstorm ideas for what would be useful to a developer who's unsure why their application is crashing, underperforming, or having other issues.

Please add to this list or leave comments below if some of this is wrong or confusing!

Broad categories of debugging questions:

• Why is my app not building?
• Why can my app not connect to external services?
• How is my app performing? How does it handle load?
• How fast does my app restart and scale?
• Does my app have any known security vulnerabilities?
• What change did I make that broke everything?

Some potential workflows/tooling (random ideas, no promise on quality):

• Easy shell into container
• Output of all deployed yaml
• Open tracing of buildpack/pipeline
• Tracking open/read of service files (probably not a good idea, but could give an idea of what's actually being used)
• Telepresence (already know this is problematic, just listing for completeness)
• Prometheus collectors
• OpenTelemetry/Jaeger (Auto instrumentation?)
• Structured logging collection
• linkerd tap https://github.com/adleong/tapshark
• linkerd routing metrics https://linkerd.io/2.11/tasks/getting-per-route-metrics/
• Could we autogenerate openapi spec based on linkerd data?
• code auditing in pipeline
• automated testing in pipeline (testpacks as a derivative of buildpacks?)
• A/B traffic mirroring
• read openapi spec to validate conforming
• Trivy in line with deploy?
• Load generator
• Fuzzing
• Network latency generator
• Opni integrations?
• lockc in "permissive mode"?
• Metrics to chargeback data? KubeCost?

[agracey]

IMO, these are required as they are expectations for any dev tool:

• Shell into workload container from epinio cli
• Port forwarding to allow attaching remote debugger
• Documenting buildpack flags to preserve debug symbols (if these are the same across languages, maybe add a --debug flag?)

My short list for additional features:

• Service Marketplace
• OpenTelemetry Collector as sidecar
• Traffic Tapping from Linkerd (and gracefully fail if no Linkerd)
• Shell into workload container via API to allow VS Code integration
• Faster build times (if possible?)
• epinio app watch that streams deltas (debounced) and triggers rebuild continuously

[agracey]

I'd really like to use ephemeral containers for the shell access but since it's beta only in 1.23, we might need to change over in a year or so.

[agracey]

Added #1098 and #1099

[jimmykarily]

Regarding OpenTelemetry Collector as sidecar, if I understand correctly by reading here, the collector is simply the thing that forwards the opentelemetry data to some backend. But what produces the openentelemetry data? What kind of metrics are we interested in? Should we just setup the collector sidecar and leave the rest to the application developer (e.g. if the application doesn't send any telemetry data to the collector, then the collector is not used). I'm not very familiar with opentelemetry, I'm simply trying to understand what we need by reading the docs. Have I missed the point?

[agracey]

Until we have the ability to build buildpacks that autoinstrument code, I would expect the responsibility of producing metrics and traces to be the responsibility of the application developer. What I'm more concerned with is the sidecar and its configuration (as well as optionally installing the linkerd jaeger plugin to consume traces)

Separately, it would be nice to do this for our control plane as well. But that's another ticket for the future.

[jimmykarily]

After discussing with @agracey we decided to postpone the work on this sidecar because it may be better to solve this with the "app to helm chart" idea. With that implemented, any sidecar could be injected by the (human) operator and epinio wouldn't have to care. If we don't do the helm chart thing, we will discuss this sidecar again.

[agracey]

Should we add that ticket (#361) to the milestone as well?

[jimmykarily]

We already got the service marketplace in and I feel the milestone grows too big. The focus of this milestone is developer experience and this feature is more for dev-ops than it is for devs. I would prefer if we didn't.

[jimmykarily]

I just tried to use trivy against an application image in epinio:

• Found the registry credentials in the registry-creds secret
• Exported env variables for trivy (docs):

export TRIVY_USERNAME=admin
export TRIVY_PASSWORD=your_password_here
export TRIVY_NON_SSL=true

• Pushed an app with epinio and copied the image url from the logs
• Run trivy with:

trivy image epinio-registry.10.86.4.38.omg.howdoi.website/apps/workspace-sample:6417241fc35728fb

I got a lot of MEDIUM and LOW warnings. There is a lot of output. I wonder what would be a nice way to present that to the user. Maybe this information could be returned from new endpoint? In the UI is could appear as a separate page or section.

trivy.txt

[agracey]

I would be hesitant to build trivy in to Epinio directly since Rancher will be integrating NeuVector which already has image scanning as well. Let me ask around and see what the plans are there and if we should factor them in or not