Report drifting close time on local node #4101
Open
+37
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marta-lokhova
force-pushed
the
clock_drift_reporting
branch
from
b4d6657
to
ea69e1a
Compare
marta-lokhova
force-pushed
the
clock_drift_reporting
branch
from
ea69e1a
to
94b1721
Compare
SirTyson
approved these changes
Jan 3, 2024
| @@ -1323,6 +1328,29 @@ HerderImpl::triggerNextLedger(uint32_t ledgerSeqToTrigger, | |||
| if (nextCloseTime <= lcl.header.scpValue.closeTime) | |||
| { | |||
| nextCloseTime = lcl.header.scpValue.closeTime + 1; | |||
| // LCL close time is when _previous_ value was nominated (normally | |||
There was a problem hiding this comment.
What is the purpose of BAD_CLOCK_PER_LEDGER_RATE_TRIGGER? Is the idea that a healthy node will occasionally drift by a second or two and be slightly behind the rest of the network? If so, why not just check that the network and local clocks are within some allowed drift window?
| @@ -1323,6 +1328,29 @@ HerderImpl::triggerNextLedger(uint32_t ledgerSeqToTrigger, | |||
| if (nextCloseTime <= lcl.header.scpValue.closeTime) | |||
There was a problem hiding this comment.
Is there any particular reason we only check if a clock is drifting behind and not drifting ahead?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
potentially resolves #1815
This change aims to report a scenario where that we frequently observe on pubnet. Often one or two validators will have a clock that is lagging, which will cause them to adjust their close time during nomination. This change adds a new metric to track frequency of such occurrences and warn operators it it happens consistently over 15 minutes (we want to report these things overtime to avoid noise).