iot: IoT/CustomResource attribute error: Vendor response doesn't contain endpointAddress

[grant-d]

Describe the bug

If I upgrade cdk & cdk-lib to 2.138.0 from 2.137.0, the following error occurs at deployment.
If I rollback, the error goes away.

CustomResource attribute error: Vendor response doesn't contain endpointAddress attribute in object ...|...IoTEndpoint

This was previously working for quite some time.

The specific resource is an AwsCustomResource related to IoT.
(I could not find a way to get the endpoint using aws-cdk-lib/aws-iot, so had to use a custom resource)

Code repro below.

Expected Behavior

Deployment succeeds

Current Behavior

Deployment fails with the following error:

8:34:21 AM | UPDATE_FAILED        | AWS::SSM::Parameter             | fooservice/Stage2...IotEndpointAddress
CustomResource attribute error: Vendor response doesn't contain endpointAddress attribute in object
arn:aws:cloudformation:eu-central-1:<account>:stack/<...>|...IoTEndpoint...

Reproduction Steps

    import * as cus from 'aws-cdk-lib/custom-resources'

    // https://stackoverflow.com/a/77092236
    const getIoTEndpoint = new cus.AwsCustomResource(this, 'IoTEndpoint', {
      onCreate: {
        service: 'Iot',
        action: 'describeEndpoint',
        // I am guessing the failure is on the following line (and/or below)
        physicalResourceId: cus.PhysicalResourceId.fromResponse('endpointAddress'),
        parameters: {
          endpointType: 'iot:Data-ATS'
        }
      },
      policy: cus.AwsCustomResourcePolicy.fromSdkCalls({ resources: cus.AwsCustomResourcePolicy.ANY_RESOURCE })
    })

    // I am guessing the failure is on the following line (and/or above)
    const IOT_ENDPOINT = getIoTEndpoint.getResponseField('endpointAddress') // eg, abc...xyz-ats.iot.us-west-2.amazonaws.com

Possible Solution

Rollback cdk-lib to 2.137.0

Additional Information/Context

No response

CDK CLI Version

2.119.0 (build 0392e71)

Framework Version

2.138.0

Node.js Version

v20.7.0

OS

MacOS

Language

TypeScript

Language Version

5.4.5

Other information

No response

[grant-d]

Related (I think): #25283
Though, once again, this issue only shows up in 2.138.0 and goes away in 2.137.0

Note that in both versions, the generated CFTs seem to look exactly the same.
And - per linked issue - neither has an Update property:

{
   "Type": "Custom::AWS",
   "Properties": {
    "ServiceToken": {
     "Fn::GetAtt": [
      "<redacted>",
      "Arn"
     ]
    },
    "Create": "{\"service\":\"Iot\",\"action\":\"describeEndpoint\",\"physicalResourceId\":{\"responsePath\":\"endpointAddress\"},\"parameters\":{\"endpointType\":\"iot:Data-ATS\"},\"logApiResponseData\":true}",
    "InstallLatestAwsSdk": false
   },
   "DependsOn": [
    "<redacted>"
   ],
   "UpdateReplacePolicy": "Delete",
   "DeletionPolicy": "Delete",
   "Metadata": {
    "aws:cdk:path": "<redacted>/IoTEndpoint/Resource/Default"
   }
  }

[pahud]

Looks like your response does not contain the endpointAddress.

const IOT_ENDPOINT = getIoTEndpoint.getResponseField('endpointAddress') 

Are you able to check the cloudwatch logs for the custom resource and see the full response object of that API call? I believe the full response object should be in the log.

[pahud]

And, can you check if this issue exists in 2.139.0 as it fixed some custom resource issues.

[grant-d]

Thanks @pahud . I deployed 2.139.0 and it also fails. Once again, rolling back to 137 works.
I will check logs and get back to you

[markusl]

@pahud I can also confirm that there is a breaking change in AwsCustomResource after 2.137.0.

export const getGroupId = (scope: Construct, identityStoreId: string, group: string, logGroup: logs.LogGroup) =>
  new cr.AwsCustomResource(scope, `getGroupId-${group}`, {
    onCreate: {
      service: 'IdentityStore',
      action: 'getGroupId',
      parameters: {
        IdentityStoreId: identityStoreId,
        AlternateIdentifier: {
          UniqueAttribute: {
            AttributePath: 'DisplayName',
            AttributeValue: group,
          },
        },
      },
      physicalResourceId: cr.PhysicalResourceId.fromResponse('GroupId'),
    },
    policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
      resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
    }),
    logGroup,
  });

The error message is CustomResource attribute error: Vendor response doesn't contain GroupId attribute in object arn:aws:cloudformation:us-east-8:999999999999:stack/stack-name. The GroupId does exist and is returned from the API when doing the API call.

Can you please have a look at this since it is quite critical and would block us from deploying AWS SSO Assignments?

Regards,
Markus

[markusl]

Is this related to the issue now that a new attribute logApiResponseData has been added?

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/cluster-resource.ts

Line 95 in c949076

// resource. Otherwise, CloudFormation will error with "Vendor response

[pahud]

Bumping this to p1 and I have escalated this to the team.

[colifran]

@markusl that would only be for the EKS cluster-resource. AwsCustomResource has its own custom resource it creates in its constructor:

aws-cdk/packages/aws-cdk-lib/custom-resources/lib/aws-custom-resource/aws-custom-resource.ts

Line 520 in fd3a5e3

this.customResource = new cdk.CustomResource(this, 'Resource', {

[pahud]

Hi @grant-d

I can deploy the following code in 2.138.0 and 2.139.0 with exactly the same output as below:

Outputs:
dummy-stack2.EndpointOutput = a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const getIoTEndpoint = new cr.AwsCustomResource(this, 'IoTEndpoint', {
      onCreate: {
        service: 'Iot',
        action: 'describeEndpoint',
        physicalResourceId: cr.PhysicalResourceId.fromResponse('endpointAddress'),
        parameters: {
          endpointType: 'iot:Data-ATS'
        }
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({ resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE })
    })

    const IOT_ENDPOINT = getIoTEndpoint.getResponseField('endpointAddress')
    new CfnOutput(this, 'EndpointOutput', { value: IOT_ENDPOINT });
  }
}

I can't reproduce this issue on my end.

This is my CloudWatch Logs

[
    "INIT_START Runtime Version: nodejs:18.v28\tRuntime Version ARN: arn:aws:lambda:us-east-1::runtime:b475b23763329123d9e6f79f51886d0e1054f727f5b90ec945fcb2a3ec09afdd\n",
    "START RequestId: 5ab66fdd-e0d0-4024-aa7d-b2b3456df051 Version: $LATEST\n",
    "2024-04-30T15:43:21.937Z\t5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tINFO\t{\"RequestType\":\"Create\",\"ServiceToken\":\"arn:aws:lambda:us-east-1:deducted:function:dummy-stack2-AWS679f53fac002430cb0da5b7982bd22872D-I2hig52uIdkt\",\"ResponseURL\":\"...\",\"StackId\":\"arn:aws:cloudformation:us-east-1:deducted:stack/dummy-stack2/435fcf60-0708-11ef-85cf-0ee587ffad51\",\"RequestId\":\"36187d1d-4add-4601-9764-b850ee234636\",\"LogicalResourceId\":\"IoTEndpoint9F0B923E\",\"ResourceType\":\"Custom::AWS\",\"ResourceProperties\":{\"ServiceToken\":\"arn:aws:lambda:us-east-1:deducted:function:dummy-stack2-AWS679f53fac002430cb0da5b7982bd22872D-I2hig52uIdkt\",\"InstallLatestAwsSdk\":\"false\",\"Create\":{\"service\":\"Iot\",\"action\":\"describeEndpoint\",\"physicalResourceId\":{\"responsePath\":\"endpointAddress\"},\"parameters\":{\"endpointType\":\"iot:Data-ATS\"},\"logApiResponseData\":true}}}\n",
    "2024-04-30T15:43:23.658Z\t5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tINFO\tAPI response { endpointAddress: 'a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com' }\n",
    "2024-04-30T15:43:23.696Z\t5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tINFO\tResponding {\"Status\":\"SUCCESS\",\"Reason\":\"OK\",\"PhysicalResourceId\":\"a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com\",\"StackId\":\"arn:aws:cloudformation:us-east-1:deducted:stack/dummy-stack2/435fcf60-0708-11ef-85cf-0ee587ffad51\",\"RequestId\":\"36187d1d-4add-4601-9764-b850ee234636\",\"LogicalResourceId\":\"IoTEndpoint9F0B923E\",\"NoEcho\":false,\"Data\":{\"region\":\"us-east-1\",\"endpointAddress\":\"a2we3h0d2g8ljn-ats.iot.us-east-1.amazonaws.com\"}}\n",
    "END RequestId: 5ab66fdd-e0d0-4024-aa7d-b2b3456df051\n",
    "REPORT RequestId: 5ab66fdd-e0d0-4024-aa7d-b2b3456df051\tDuration: 5154.74 ms\tBilled Duration: 5155 ms\tMemory Size: 128 MB\tMax Memory Used: 97 MB\tInit Duration: 181.34 ms\t\n"
]

Can you try my code snippets above and see if it works for you?

[pahud]

Looking at your provided code, I would not recommend specifying physicalResourceId like this because in this case your physicalResourceId would be subject to change from the response you receive and things may break unexpectedly if the SDK response does not contain that field in some edge cases.

physicalResourceId: cr.PhysicalResourceId.fromResponse('endpointAddress'),

Instead, I would recommend this statically.

physicalResourceId: cr.PhysicalResourceId.of('IoTEndpoint'),

Let me know if it works for you.

[pahud]

@markusl I was not able to execute getGroupId in my account but I have similar code

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const groupId = new cr.AwsCustomResource(this, 'getGroupId', {
      onCreate: {
        service: 'IdentityStore',
        action: 'describeGroup',
        parameters: {
          IdentityStoreId,
          GroupId,
        },
        physicalResourceId: cr.PhysicalResourceId.fromResponse('GroupId'),
      },
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
        resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
      }),
    });

    const attrGroupId =  groupId.getResponseField('GroupId');
    new CfnOutput(this, 'GroupIdOutput', { value: attrGroupId });
  }
}

And I was able to get the GroupId from its response. And this works in 2.137.0, 2.138.0 and 2.139.0.

Are you able to successfully run my snippet above?

[pahud]

We should decouple the physicalResourceId from the API response as describe here to avoid potential error:

#30012 (comment)

[markusl]

In our case, with the help of AWS support, we were able to identify the problem, which was a missing onUpdate handler, which is required when performing a CDK version upgrade due to the added property.

We are now using the same handler for onCreate and onUpdate which returns the group name properly.

// AWS Custom Resource to dynamically fetch GroupId based on group name
export const getGroupId = (scope: Construct, identityStoreId: string, group: string, logGroup: logs.LogGroup) => new cr.AwsCustomResource(scope, `getGroupId-${group}`, {
  onCreate: getGroupIdSdkCall(identityStoreId, group),
  onUpdate: getGroupIdSdkCall(identityStoreId, group),
  policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
    resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
  }),
  logGroup,
});

I'm not sure if this is a "breaking change" or not, but I would suggest highlighting this properly in the changelog that the CDK version upgrade will cause an update to all Custom Resources.

[colifran]

@markusl Thanks for the follow-up and I agree, this feels like a documentation miss on our end. From the perspective of the custom resource it saw an update and didn't make an API call which is how it was configured. However, obviously you now don't have a groupId since your custom resource was configured to only make the API call on a create event. We'll work on improving the documentation for this.

[grant-d]

Thanks @pahud , I have just tested again with both versions (and @140) and same issue.

I am confounded why it does not repro, there is no difference in our code (my snippet was the real implementation). I have also used your suggested PhysicalResourceId idiom and it makes no difference - still fails. I will report back on CDK logs forgive my ignorance - I added a log group to the custom resource lambda but after failure it contains no log streams. Where are you looking for said deployment logs?

Edit. Log config was fine but it never even got to it before crashing the lambda.
Found a fix but makes no sense:

• I deleted (commented-out) the custom-resource entirely from the stack & deployed
• Upgraded CDK (2.140)
• Added (uncommented) exact same code back (except for your suggestion), deployed again
• All is good now, deployment successful

I am happy to close this issue, but will leave that up to you since there are similar issues reported in this thread.

[blimmer]

This caused a bit of a fire drill for me today, too. I'm using a Custom Resource to work around the lack of native CloudFormation support of AWS::IdentityStore::User.

Previously, I had this code:

const user = new AwsCustomResource(this, `${emailAddress}User`, {
  resourceType: 'Custom::IdentityStoreUser',
  onCreate: {
    service: '@aws-sdk/client-identitystore',
    action: 'CreateUserCommand',
    parameters: {
      IdentityStoreId: identityStoreId,
      UserName: emailAddress,
      DisplayName: userInfo.name,
      Emails: [
        {
          Value: emailAddress,
          Type: 'Work',
          Primary: true,
        },
      ],
      Name: {
        GivenName: firstName,
        FamilyName: lastName,
      },
    },
    physicalResourceId: PhysicalResourceId.fromResponse('UserId'),
  },
  onDelete: {
    service: '@aws-sdk/client-identitystore',
    action: 'DeleteUserCommand',
    parameters: {
      IdentityStoreId: identityStoreId,
      UserId: new PhysicalResourceIdReference(),
    },
  },
  policy: AwsCustomResourcePolicy.fromStatements([
    new PolicyStatement({
      actions: ['identitystore:CreateUser', 'identityStore:UpdateUser', 'identitystore:DeleteUser'],
      resources: [identityStoreArn, 'arn:aws:identitystore:::user/*'],
    }),
  ]),
});

// Add the user to the specified groups
userInfo.groupMembership.forEach((groupName) => {
  new CfnGroupMembership(this, `${emailAddress}${groupName}Membership`, {
    identityStoreId,
    groupId: groups[groupName].attrGroupId,
    memberId: { userId: user.getResponseField('UserId') },
  });
});

When I upgraded from 2.136.0 => 2.140.0, I started getting errors that looked like this:

Vendor response doesn't contain UserId attribute in object

The fix for me was changing the getResponseField code to use the Physical ID I'm setting on the user object:

// Add the user to the specified groups
userInfo.groupMembership.forEach((groupName) => {
  new CfnGroupMembership(this, `${emailAddress}${groupName}Membership`, {
    identityStoreId,
    groupId: groups[groupName].attrGroupId,
    memberId: { userId: (user.node.defaultChild as CustomResource).ref },  // <-- this is the change
  });
});

The documentation mentions that I shouldn't use getResponseField in my case (without an update call), but I missed it the first time:

AwsCustomResource.getResponseField() and .getResponseFieldReference() will not work if the Create and Update APIs don't consistently return the same fields.

I also added an onUpdate that calls UpdateUserCommand for good measure.