Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alerts triggered status is not persistent #679

Closed
macmoritz opened this issue Feb 17, 2024 · 3 comments · Fixed by #764
Closed

Alerts triggered status is not persistent #679

macmoritz opened this issue Feb 17, 2024 · 3 comments · Fixed by #764
Labels
area/alerting Related to alerting area/storage Related to storage bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed

Comments

@macmoritz
Copy link
Contributor

Describe the bug

The triggered status of a monitor is only momentary. After a restart of gatus the information is gone.

Such restarts are daily business in cloud environments.

What do you see?

No response

What do you expect to see?

Gatus should only send one alert for a failing monitor.

List the steps that must be taken to reproduce this issue

  1. gatus starts
  2. after some time a monitored service fails
  3. gatus sends an alert after checking if alert was sended before
  4. gatus will not send more notifications for this monitor, which is correct (log: [watchdog][handleAlertsToTrigger] Sending matrix alert because alert for endpoint=my-endpoint='' has been TRIGGERED)
  5. gatus restarts
  6. gatus will send an alert for the monitor again

Version

No response

Additional information

Fix could be to use the events from the database for checking if an alert was already sended.
@macmoritz macmoritz added the bug Something isn't working label Feb 17, 2024
@TwiN TwiN added help wanted Extra attention is needed good first issue Good for newcomers area/alerting Related to alerting area/storage Related to storage labels Feb 17, 2024
@TwiN
Copy link
Owner

TwiN commented Feb 17, 2024

Yes, you're right. It's also something I've been wanting to take care of for a while.

@TwiN TwiN mentioned this issue Apr 7, 2024
Merged
2 tasks
@ser
Copy link

ser commented Apr 30, 2024 •
edited

I would also add something like UUID of the sent alert and keep the same UUID when the alert gets resolved or sending is repeated by any reason. That UUID should be available in the alert templating.

@TwiN TwiN pinned this issue May 11, 2024
TwiN added a commit that referenced this issue May 15, 2024
@TwiN
feat(alerting): Persist triggered alerts across application restart
bd841a0 
Fixes #679
@TwiN TwiN mentioned this issue May 15, 2024
Merged
2 tasks
@TwiN
Copy link
Owner

TwiN commented May 15, 2024

PR is up @ #764 but further testing is necessary

@TwiN TwiN closed this as completed in #764 May 16, 2024
TwiN added a commit that referenced this issue May 16, 2024
@TwiN
feat(alerting): Persist triggered alerts across application restart (#…
f2c5f59 
…764)

* feat(alerting): Persist triggered alerts across application restart

Fixes #679

* test(alerting): Add numerous tests related to alerts
TwiN added a commit that referenced this issue May 17, 2024
@TwiN
fix(alerting): Make sure alert is triggered before attempting to pers…
2a94f76 
…ist it to the database

Related to #764 and #679
@TwiN TwiN unpinned this issue Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/alerting Related to alerting area/storage Related to storage bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(alerting): Persist triggered alerts across application restart TwiN/gatus
3 participants
@ser @TwiN @macmoritz