[Bug]: WriteTooOldError: write for key

[pschiffe]

Preflight Checklist

• I could not find a solution in the documentation, the existing issues or discussions
• I have joined the ZITADEL chat

Environment

Self-hosted

Version

2.51.4

Database

CockroachDB

Database Version

23.2.5

Describe the problem caused by this bug

Hello, Zitadel is spamming me with the following errors:

{"caller":"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:631","error":"ID=CRDB-pKtsr Message=exec failed Parent=(ERROR: restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/123/1/\"255905483095801859\"/\"255816223223185411\"/\"225367213287145475\"/0 at timestamp 1715787882.440995219,0 too old; must write at or above 1715787882.509992034,1: \"sql txn\" meta={id=e5efa617 key=/Table/231/1/\"auth.user_sessions\"/\"225367213287145475\"/0 iso=Serializable pri=0.03755661 epo=0 ts=1715787882.509992034,1 min=1715787882.440995219,0 seq=1} lock=true stat=PENDING rts=1715787882.440995219,0 wto=false gul=1715787882.940995219,0 (SQLSTATE 40001))","level":"error","msg":"statement execution failed","projection":"auth.user_sessions","time":"2024-05-15T15:44:42Z"}
{"aggregate":"255816223223185411","caller":"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/failed_event.go:53","error":"ID=CRDB-Unnex Message=unable to update failure count Parent=(ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02))","instance":"225367213287145475","level":"warning","msg":"unable to get failure count","projection":"auth.user_sessions","sequence":38,"time":"2024-05-15T15:44:42Z"}
{"caller":"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/state.go:96","error":"ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02)","level":"warning","msg":"unable to update state","projection":"auth.user_sessions","time":"2024-05-15T15:44:42Z"}
{"caller":"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:416","error":"ID=V2-WF23g2 Message=unable to update state Parent=(ERROR: current transaction is aborted, commands ignored until end of transaction block (SQLSTATE 25P02))","level":"info","msg":"process events failed","projection":"auth.user_sessions","time":"2024-05-15T15:44:42Z"}

It started recently, there was no configuration changes, just updates to Zitadel or Cockroachdb, not sure what triggered it. Zitadel seems to work correctly even with the errors.

To reproduce

Not sure.

Screenshots

No response

Expected behavior

No response

Operating System

No response

Relevant Configuration

No response

Additional Context

No response

[pschiffe]

I'm seeing 10 000 of these logs every 5 minutes, even with version 2.52.0:

# docker logs zitadel --since 5m 2>&1 | wc -l
10881

Any advice how to fix this?

[livio-a]

will be fixed by: #7982
in the meantime you can set the flag manually as described in the PR

[pschiffe]

This bug was fixed by PR #7982 and released in version v2.53.0