Heartbeats limited to QUERY_LIMIT (50)

[dvessey-fes]

Issue Summary
If alerta has more than 50 active heartbeats, then alerta heartbeats --alert may miss some expired/stale heartbeats

Environment

• OS: Linux

• API version: 8.4.1, but appears to be the same in latest/master

• Deployment: self hosted

• For self-hosted, WSGI environment: nginx/uwsgi

• Database: Postgres

• Server config:
  Auth enabled? Yes
  Auth provider?Basic
  Customer views? No
  (or provide the /config endpoint output)

• web UI version: NA I don't think

• CLI version: NA I don't think

To Reproduce
Steps to reproduce the behavior:

1. Create 55 hearbeats
2. List form CLI with alerta heartbeats - wait till one of the ones that doesn't show there expires
3. Run alerta heartbeats --alert - see that it doesn't alert on the expired heartbeat(s) that are no longer visibile

Expected behavior
We should get heartbeat alerts for all heartbeats, no matter the limit

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I started to dig through the code for alerta and python-alerta-client. As far as I can tell the limit of 50 comes from the default settings.py/QUERY_LIMIT. I could just increase this but that seems like it'd just be moving the goal post.

I guess part of this report is really a question - is there any technical reason to limit the number of heartbeats in the system? If yes, then I suspect that limit should be higher than 50, and alerta heartbeats --alert should page through all of them?

If that's the case please let me know, I can take a stab at a small PR to page through heartbeats.

Cheers!

NOTE: Please provide as much information about your issue as possible.
Failure to provide basic details about your specific environment make
it impossible to know if an issue has already been fixed, can delay a
response and may result in your issue being closed without a resolution.

[dvessey-fes]

Ah looks like this was already raised in #1281 and #1484