Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Somehow I was able to give 2 LGTMs as an API Owner #3692

Open
domenic opened this issue Mar 6, 2024 · 4 comments
Open

Somehow I was able to give 2 LGTMs as an API Owner #3692

domenic opened this issue Mar 6, 2024 · 4 comments
Labels

Comments

@domenic
Copy link

domenic commented Mar 6, 2024

https://chromestatus.com/feature/5189728691290112

image

Probably related, I have no memories of actually using the Chrome Status tool to give these LGTMs, and they don't show up in the comments log.

@domenic domenic added the bug label Mar 6, 2024
@jrobbins
Copy link
Collaborator

jrobbins commented Mar 6, 2024

I have a lead on this bug.

Your message to blink-dev listed blink-dev in both the To: line and the Cc: line. ChromeStatus got two copies of that email and detected your "LGTM2" twice.
https://groups.google.com/a/chromium.org/g/blink-dev/c/wRRCnNLCZs0/m/RMJ7IMhwAQAJ

The two emails were processed by ChromeStatus a second apart. If it is a race condition, we probably need to specify a read-consistency parameter on our datastore query, or it might be just failure to check for an existing vote. I'll try to fix it now.

@jrobbins
Copy link
Collaborator

jrobbins commented Mar 6, 2024

Actually, the tasks finished 1 second apart, but the two votes were written to datastore just 4ms apart. So, this seems like a straight race condition. We already use strong read consistency. I'll brainstorm with my team on what other ways we can avoid this problem.

@jrobbins
Copy link
Collaborator

jrobbins commented Mar 6, 2024

Actually, the tasks started almost exactly 1 second apart. That is consistently what I see when I try to repro the problem on our staging server. That 1s separation is because we specify a rate of 1/s in queues.yaml.

So, the first task started, it somehow got delayed for exactly 1s, and then it got into a race with the second task. From the logs:

> INFO 2024-03-05T09:11:00.983542Z In IntentEmailHandler: From addr: 'domenic@chromium.org' ...
??? What happened here and why did it take 1 whole second?
> INFO 2024-03-05T09:11:01.115760Z intent_thread_url was already set to ...
> INFO 2024-03-05T09:11:01.347828Z found LGTM
> INFO 2024-03-05T09:11:01.085133Z In IntentEmailHandler: From addr: 'domenic@chromium.org' ...
> INFO 2024-03-05T09:11:01.209873Z intent_thread_url was already set to ...
> INFO 2024-03-05T09:11:01.347243Z found LGTM

This is a race condition, but because of the 1/s rate limit, I think it is not very likely to happen. In fact, I failed to repro the problem.

@jrobbins
Copy link
Collaborator

jrobbins commented Mar 6, 2024

I deleted the duplicate vote on that review gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants