Latency/Response Spike Alerting

[nzurku]

We have found it to be relevant and easiest if Cloudprober could detect a spike in latency or response times by itself for alerting.

While the logic/configurables could be debated, something that came to my mind is this:

Rule: if Latency/Response 25% Greater
Check Range: 5 vs 25, compare 5 most recent checks to the previous 25 checks (excluding the most recent 5)
Then: Trigger alerting for a spike in latency/response, raise a metric of this spike/increase to be 1/yes.

[manugarg]

I think this is kind of interesting. We could add another condition type, say "spike" that would keep average of latencies over time and fire an alert if latency is more than the average by a certain percentage. It will require careful implementation though, I'll think more about it.