Can DynamoDB TTL behavior "break" @aws-lambda-powertools/idempotency?

[HaaLeo]

I have a concern about DynamoDB's TTL affecting the behavior of the powertool's idempotency package.

The scenario

According to dynamodb's TTL documentation an entry can still remain in the DB and occur in queries although it has already been expired:

Items that have expired, but haven’t yet been deleted by TTL, still appear in reads, queries, and scans. If you do not want expired items in the result set, you must filter them out. To do this, use a filter expression that returns only items where the Time to Live expiration value is greater than the current time in epoch format. For more information, see Filter expressions for scan.

The Concern

That raises the question whether the idempotency package might retrieve an already expired dynamoDB entry and returns the safed response instead of invoking the original function instead? Or takes the package care of that TTL behavior already?

I appreciate any hint on this :)

Thx
Leo

[dreamorosi]

Hi @HaaLeo, thank you for opening a question in our repository!

Even though it's true that DynamoDB doesn't necessarily evict items as soon as their TTL has expired, Powertools is able to handle this case and their presence in the table does not prevent the Idempotency utility to detect that the operation should be in fact processed.

When processing a new payload the Idempotency utility performs an optimistic write and tries to save the record as "in progress". The write operation uses a condition expression that includes the following statements (source):

• The attributes doesn't exist (this is a never-seen-before request or request whose record has already been evicted)
• The item exists, but its expiry attribute (aka TTL) is in the past (this is a request that has expired, but that DynamoDB hasn't evicted yet - this is the case you're referring to)
• The attribute exists, its status is "in progress", but the in-progress-expiry timestamp is in the past (this a request that has timed out during execution, and thus safe to process again).

If any of these conditions (concatenated by OR) evaluates to true, then the operation is considered as "new" and your function is called as if there was no previous record.

The request flow for this specific case is described in this section of the docs: https://docs.powertools.aws.dev/lambda/typescript/latest/utilities/idempotency/#expired-idempotency-records

[HaaLeo]

Thx @dreamorosi for your response. Although, I've skimmed through the code I obviously missed that part.
Thank you for pointing me to the right direction.👍🏼