Add profilez server endpoint

[neilalexander]

This PR adds a profilez endpoint to the system account, which accepts two parameters: profile for the name of the profile (i.e. allocs, block, goroutine, heap, mutex, threadcreate) and debug for the return format (defaults to 0, which is usually gzipped for go tool pprof).

It will only be possible to query this endpoint if PROF_PORT is already enabled, otherwise it will return a Profiling is not enabled error.

Signed-off-by: Neil Twigg neil@nats.io

/cc @nats-io/core

[ripienaar]

We also have configuration option in the server to enable it at runtime and when embedded. This kind of thing I think should be an opt-in as it can have runtime performance impact on the servers

[neilalexander]

This kind of thing I think should be an opt-in as it can have runtime performance impact on the servers

So far the only thing that StartProfiler does that has any overhead is to call runtime.SetBlockProfileRate, which profilez does not do admittedly, so maybe we can just guard off the block profile unless StartProfiler has been called.

There's no runtime overhead on things like allocs, goroutine, heap, threadcreate as they just are internally extracting statistics out of the runtime package.

[ripienaar]

There's certainly some cost associated with gathering a profile though - if its same as when you hit the normal pprof end points the go team aims to keep the impact aroun 5%, its still an impact though and a potential vector for abuse that someone might want to turn off.

[neilalexander]

I should have probably noted in the original PR that it isn't actually possible to collect a CPU profile via profilez right now (like you would with /debug/pprof/profile) because the pprof package registers that straight onto DefaultServeMux and not into the profiles struct, so the only accessible profiles are these.

I can make it opt-in, although it doesn't look as though the monitoring port actually has a configuration section of its own, so I guess I could do one of the following:

1. Add a http_profiling option, to control whether or not it should be exposed on both the monitoring port and the system account;
2. Only expose it on the system account only (and not the HTTP port) when PROF_PORT is already enabled?

[ripienaar]

2nd options sounds good - the current /debug/pprof stuff is nice cos its just exactly what go tool pprof wants for real time debugging etc. so for HTTP I think its great as is, then just additionally expose on system account once opted in is great

[derekcollison]

Use EMPTY, old pattern but want to stay consistent.

[derekcollison]

EMPTY

[derekcollison]

Please do a rebase onto dev vs merge.

[derekcollison]

LGTM!