(aws-cdk-lib.aws_s3_notifications): (Configuration is ambigously defined even through prefixes are not overlapping)

[thenamanpatwari]

Describe the bug

The S3 notification module is throwing an configuration is ambiguously defined error even though the prefixes do not overlap. This issue is resolved if I remove all currently defined notifications and re-add them along with the new one.

Example code

    //2D Bounding Box Upload Notification
    this.privateBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(props.eventLambda), {
      // Declared in s3_generic_map.py
      prefix: "bounding_box_2d/",
    });

    //KV Rule Engine Notification
    this.privateBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(props.eventLambda), {
      // Declared in s3_generic_map.py
      prefix: "kv_rule_engine/input/",
    });

Expected Behavior

I expect the relevant event notifications to be created on the relevant S3 bucket

Current Behavior

CloudFromation is throwing an error

notification configuration
Traceback (most recent call last):
  File "/var/task/index.py", line 24, in handler
    s3.put_bucket_notification_configuration(Bucket=props["BucketName"], NotificationConfiguration=config)
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lang/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type

Reproduction Steps

1. Create a single event notification with some prefix on an S3 bucket
2. Deploy changes
3. Create another event notification on the same S3 bucket but with a different prefix
4. Observe error thrown by CloudFormation

Possible Solution

No response

Additional Information/Context

Recently upgraded to "aws-cdk-lib": "^2.138.0" which might be causing this issue

CDK CLI Version

2.138.0

Framework Version

No response

Node.js Version

10.8.1

OS

Linux

Language

TypeScript

Language Version

No response

Other information

No response

[ashishdhingra]

@thenamanpatwari Good morning. Please share the following:

• If you are using Python or TypeScript to write CDK stack (I noticed code comment // Declared in s3_generic_map.py).
• Are you able to synthesize the CF template using cdk synth?
  • What does it look like with respect to event notification perspective?
  • What is the output of cdk diff?
• Share the minimal end-to-end reproduction code for us to troubleshoot the issue.

Using the below CDK TypeScript code, I was able to deploy CF stack (first deploying one event notification and then another one as part of same stack):

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3n from 'aws-cdk-lib/aws-s3-notifications'

export class TypescriptStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const lambdaFunction = new lambda.Function(this, 'TestLambdaFunction',{
      runtime: lambda.Runtime.NODEJS_18_X,
      code: lambda.Code.fromAsset('lambda'),
      handler: 'hello.handler'
    });
    const s3Bucket = new s3.Bucket(this, 'TestBucket');
    s3Bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaFunction), { prefix: "bounding_box_2d/" });
    s3Bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaFunction), { prefix: "kv_rule_engine/input/" });
  }
}

const app = new cdk.App();
new TypescriptStack(app, 'TypescriptStack');

Thereafter, when I upload an object to S3, I see no error in the CloudWatch logs for BucketNotificationsHandler created by CDK stack (runtime Python):

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                 message                                                                                  |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1714068023786 | INIT_START Runtime Version: python:3.11.v34 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:1bebe65bb4f19dd8c37baeb0be18a0b278155b42bfe12dd28480199436d18229      |
| 1714068024198 | START RequestId: e5e39f6d-4059-4db8-96f5-a486f0c128cb Version: $LATEST                                                                                                   |
| 1714068024959 | Status code: OK                                                                                                                                                          |
| 1714068024966 | END RequestId: e5e39f6d-4059-4db8-96f5-a486f0c128cb                                                                                                                      |
| 1714068024966 | REPORT RequestId: e5e39f6d-4059-4db8-96f5-a486f0c128cb Duration: 768.30 ms Billed Duration: 769 ms Memory Size: 128 MB Max Memory Used: 84 MB Init Duration: 409.95 ms   |
| 1714068091179 | START RequestId: 85d47611-dfe2-4861-98f9-f3fa18e16e76 Version: $LATEST                                                                                                   |
| 1714068091824 | Status code: OK                                                                                                                                                          |
| 1714068091830 | END RequestId: 85d47611-dfe2-4861-98f9-f3fa18e16e76                                                                                                                      |
| 1714068091830 | REPORT RequestId: 85d47611-dfe2-4861-98f9-f3fa18e16e76 Duration: 651.07 ms Billed Duration: 652 ms Memory Size: 128 MB Max Memory Used: 85 MB                            |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks,
Ashish

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[thenamanpatwari]

Thank you for your response Ashish.

1. Typescript. The .py file comment is a reference to the python code used by our lamdba functions.
2. Yes I am able to synthesize the stack. The error is thrown by CloudFormation during deployment.
3. Those are the steps I follow to recreate the issue. I do have a lot of prefixes already defined, but whenever I add a new one the deployment stage throws an error. I resolve this issue by removing all existing prefixes and adding them all back along with the new target prefix

[airmonitor]

I'm having the same issue, with the below changes

Resources
[~] Custom::S3BucketNotifications imported_source_bucket/Notifications importedresourcesimportedsourcebucketNotifications2FD58448 
 └─ [~] NotificationConfiguration
     └─ [~] .TopicConfigurations:
         └─ @@ -55,6 +55,24 @@
            [ ] },
            [ ] {
            [ ]   "Events": [
            [+]     "s3:ObjectCreated:*"
            [+]   ],
            [+]   "Filter": {
            [+]     "Key": {
            [+]       "FilterRules": [
            [+]         {
            [+]           "Name": "suffix",
            [+]           "Value": ".mpd"
            [+]         }
            [+]       ]
            [+]     }
            [+]   },
            [+]   "TopicArn": {
            [+]     "Ref": "topicconstructshobjectcopyfanout2A14F786"
            [+]   }
            [+] },
            [+] {
            [+]   "Events": [
            [ ]     "s3:ObjectRemoved:*"
            [ ]   ],
            [ ]   "Filter": {
            @@ -106,5 +124,23 @@
            [ ]     "TopicArn": {
            [ ]       "Ref": "topicconstructshobjectdeletefanout09E6ABCD"
            [ ]     }
            [+]   },
            [+]   {
            [+]     "Events": [
            [+]       "s3:ObjectRemoved:*"
            [+]     ],
            [+]     "Filter": {
            [+]       "Key": {
            [+]         "FilterRules": [
            [+]           {
            [+]             "Name": "suffix",
            [+]             "Value": ".mpd"
            [+]           }
            [+]         ]
            [+]       }
            [+]     },
            [+]     "TopicArn": {
            [+]       "Ref": "topicconstructshobjectdeletefanout09E6ABCD"
            [+]     }
            [ ]   }
            [ ] ]


✨  Number of stacks with differences: 1

cdk version 2.139.1 (build b88f959)

stack is failing with the below:

9:14:26 PM | UPDATE_FAILED        | Custom::S3BucketNotifications | importedresourcesi...ifications2FD58448
Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Co
nfiguration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.. See the details in CloudWatch Log Stream:
2024/04/30/[$LATEST]e4ef3d9576c34417988f05f45f62285a (RequestId: 96d9d9dc-b8f7-47f5-ac33-02d3df762f43)

9:14:30 PM | UPDATE_FAILED        | Custom::S3BucketNotifications | importedresourcesi...ifications2FD58448
Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Co
nfiguration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.. See the details in CloudWatch Log Stream:
2024/04/30/[$LATEST]e4ef3d9576c34417988f05f45f62285a (RequestId: a63be1f4-7d52-42fa-a3f9-0dbfa2155d67)


 ❌  sns-fan-out-stack (sns-fan-out-stack) failed: Error: The stack named sns-fan-out-stack failed to deploy: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [importedresourcesimportedsourcebucketNotifications2FD58448]. ): Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.. See the details in CloudWatch Log Stream: 2024/04/30/[$LATEST]e4ef3d9576c34417988f05f45f62285a (RequestId: 96d9d9dc-b8f7-47f5-ac33-02d3df762f43)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:433:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:436:199619)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:436:181341

 ❌ Deployment failed: Error: The stack named sns-fan-out-stack failed to deploy: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [importedresourcesimportedsourcebucketNotifications2FD58448]. ): Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.. See the details in CloudWatch Log Stream: 2024/04/30/[$LATEST]e4ef3d9576c34417988f05f45f62285a (RequestId: 96d9d9dc-b8f7-47f5-ac33-02d3df762f43)
    at FullCloudFormationDeployment.monitorDeployment (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:433:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:436:199619)
    at async /opt/homebrew/lib/node_modules/aws-cdk/lib/index.js:436:181341

The stack named sns-fan-out-stack failed to deploy: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [importedresourcesimportedsourcebucketNotifications2FD58448]. ): Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type.. See the details in CloudWatch Log Stream: 2024/04/30/[$LATEST]e4ef3d9576c34417988f05f45f62285a (RequestId: 96d9d9dc-b8f7-47f5-ac33-02d3df762f43)

[thenamanpatwari]

@airmonitor Thank you for the detailed description. Can confirm I am getting the same error.
Worst part is that the stack is left in UPDATE_ROLLBACK_FAILED state, which requires manual intervention (skip updating the S3 bucket) for the rollback to complete.

[thenamanpatwari]

@ashishdhingra any updates? This is blocking some of our critical deployments.

[ashishdhingra]

@thenamanpatwari As mentioned in the #29951 (comment), I'm unable to reproduce the issue. Please share the sample end-to-end code (including Python code for your lambda function) and any other configuration that needs to be done, to reproduce the issue. From the error in the issue description, it appears that you have overlapping prefixes for the same event type. You also mentioned that there are lot of prefixes already defined. Are these defined for same event type?

Also the error in the issue description appears to be in Lambda event handler, looks like it is reported when the Lambda is invoked per event notification, not when CDK stack is deployed. Kindly correct if I'm wrong.

Kindly note that once the CDK deployment is initiated, it is CloudFormation that takes over.

Thanks,
Ashish