Consumer group lag property is not updated when unread messages are evicted from a stream

[carlos-gaspar]

We have been expecting that the lag property (provided for stream consumer groups with command XINFO GROUPS) to be updated in case of messages that are removed from the stream before being read by the consumer group. It would be really worthy and simple to be used as metric to auto scale consumer instances.

In our use case, we use XTRIM to remove old messages from the stream, so we keep messages inside a time window (like last 2 hours).

In case of unavailability of consumers, we can reach a scenario where messages are removed from stream before being read. In this case we noticed that the lag property remains with its value unchanged.

We ran an example under extreme conditions (really short time window) and we can see that all messages were removed (and thus the stream length is 0), but, since many messages were removed before even being read by a consumer, the lag information remains greater than zero and it never resets.

`127.0.0.1:6379[10]> xinfo stream myStream

1. "length"
2. (integer) 0
3. "radix-tree-keys"
4. (integer) 0
5. "radix-tree-nodes"
6. (integer) 1
7. "last-generated-id"
8. "1670921398247-0"
9. "max-deleted-entry-id"
10. "0-0"
11. "entries-added"
12. (integer) 20000
13. "recorded-first-entry-id"
14. "0-0"
15. "groups"
16. (integer) 1
17. "first-entry"
18. (nil)
19. "last-entry"
20. (nil)
    127.0.0.1:6379[10]> xinfo groups myStream
21. 1. "name"
    2. "myGroup-1"
    3. "consumers"
    4. (integer) 51
    5. "pending"
    6. (integer) 0
    7. "last-delivered-id"
    8. "1670921398247-0"
    9. "entries-read"
22. (integer) 4634
23. "lag"
24. (integer) 15366`

We were expecting that, in cases like that, the lag property to be reset.
Is the current behavior of the lag property the expected one?
In the affirmative case, is there anything different we should do?

[itamarhaber]

Hello @carlos-gaspar - what a wonderful idea :)

Off the top of my heap, I think this can be done fairly easily.... but I might be wrong (50:50), so I'd like @guybe7 to opine on this as well.

[itamarhaber]

Ok, please disregard my previous overly enthusiastic reply. Once an unread message is deleted, there's no efficient way I can think of that will let us estimate the lag between all consumer groups to the stream's tip.

[sergio-ferreira-jscrambler]

@itamarhaber does this mean that this use case will not be supported? We were hoping that we could use the LAG property to autoscale containers automatically, but with this limitation we can't