CrossRegionExportWriter: Response object is too long.

[joshbalfour]

Describe the bug

When using cross-region references between stacks I get this error:
(The following resource(s) failed to update: [ExportsWritereuwest25F52B32F9AB3BFFD]. ): Response object is too long.

This might be because I'm exporting resources from a nested stack so the generated keys are really long.

I'm aware cross-region references are very alpha, and I have a workaround I'm going to try that basically replicates the functionality but does one resource reference at a time, not ideal as it will be much slower.

Expected Behavior

The reference to work as expected.

Current Behavior

The stack fails to deploy.

Reproduction Steps

The resource when synthed looks like this:

"ExportsWritereuwest25F52B32F9AB3BFFD": {
   "Type": "Custom::CrossRegionExportWriter",
   "Properties": {
    "ServiceToken": {
     "Fn::GetAtt": [
      "CustomCrossRegionExportWriterCustomResourceProviderHandlerD8786E8A",
      "Arn"
     ]
    },
    "WriterProps": {
     "region": "eu-west-2",
     "exports": {
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydrahydrapublicB75FC865RefEA95D68D": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2oryhydrahydrapublicB75FC865Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydraadhydraadpublic8B011CFARef0C11F132": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2oryhydraadhydraadpublic8B011CFARef"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2orykratoskratospublicBF598F48Ref61F05EEB": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2orykratoskratospublicBF598F48Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydrahydraadmin44A1DC5ARef25110A0F": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2oryhydrahydraadmin44A1DC5ARef"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydraadhydraadadmin13715108Ref123ADE30": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2oryhydraadhydraadadmin13715108Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2orykratoskratosadmin66E5C2F5RefABC2E741": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2orykratoskratosadmin66E5C2F5Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttadmingqlNestedStackadmingqlNestedStackResource7F92880AOutputscompanyproductjosh22apsoutheast2admingqlalbEF458438RefF1172A0D": {
       "Fn::GetAtt": [
        "admingqlNestedStackadmingqlNestedStackResource7F92880A",
        "Outputs.companyproductjosh22apsoutheast2admingqlalbEF458438Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryproxyNestedStackproxyNestedStackResource12C3FA79Outputscompanyproductjosh22apsoutheast2oryproxyal9F2CB0FD": {
       "Fn::GetAtt": [
        "oryNestedStackoryNestedStackResource908720F1",
        "Outputs.companyproductjosh22apsoutheast2oryproxyNestedStackproxyNestedStackResource12C3FA79Outputscompanyproductjosh22apsoutheast2oryproxyalbD89592B8Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttsamlNestedStacksamlNestedStackResource2B1E690EOutputscompanyproductjosh22apsoutheast2samlalb776F7A60Ref2A2E0028": {
       "Fn::GetAtt": [
        "samlNestedStacksamlNestedStackResource2B1E690E",
        "Outputs.companyproductjosh22apsoutheast2samlalb776F7A60Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttloginNestedStackloginNestedStackResource5E1A21F5Outputscompanyproductjosh22apsoutheast2loginalb6D8F1A33Ref5E92FD74": {
       "Fn::GetAtt": [
        "loginNestedStackloginNestedStackResource5E1A21F5",
        "Outputs.companyproductjosh22apsoutheast2loginalb6D8F1A33Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttmigrationNestedStackmigrationNestedStackResourceC0800D01Outputscompanyproductjosh22apsoutheast2migrationalb2B1D1A53Ref0B868277": {
       "Fn::GetAtt": [
        "migrationNestedStackmigrationNestedStackResourceC0800D01",
        "Outputs.companyproductjosh22apsoutheast2migrationalb2B1D1A53Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttactivedirectoryfederationNestedStackactivedirectoryfederationNestedStackResourceF6043CF6Outputscompanyproductjosh22apsoutheast2activedirectoryfederationalb6E7DB668Ref1F7C174D": {
       "Fn::GetAtt": [
        "activedirectoryfederationNestedStackactivedirectoryfederationNestedStackResourceF6043CF6",
        "Outputs.companyproductjosh22apsoutheast2activedirectoryfederationalb6E7DB668Ref"
       ]
      },
      "/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttsamlfederationNestedStacksamlfederationNestedStackResourceF9BAFFEBOutputscompanyproductjosh22apsoutheast2samlfederationalb8BC22212Ref5884B1DC": {
       "Fn::GetAtt": [
        "samlfederationNestedStacksamlfederationNestedStackResourceF9BAFFEB",
        "Outputs.companyproductjosh22apsoutheast2samlfederationalb8BC22212Ref"
       ]
      }
     }
    }
   },
   "UpdateReplacePolicy": "Delete",
   "DeletionPolicy": "Delete",
   "Metadata": {
    "aws:cdk:path": "company-product-josh22/ap-southeast-2/ExportsWritereuwest25F52B32F/Resource/Default"
   }
  },

Possible Solution

Maybe using outputPaths might help, but if the key length is what's causing the issue I can see how this wouldn't necessarily help.

Additional Information/Context

No response

CDK CLI Version

2.63.0

Framework Version

No response

Node.js Version

18

OS

alpine

Language

Typescript

Language Version

No response

Other information

No response

[joshbalfour]

The custom resource logs this, I think the Response object is too long. is due to the response object being so large because of the Reason listing out the keys in the actual error that occurred.
The actual error is caused by SSM being asked to delete too many parameters in one request where the max number is 10. I'll see if batching the request to SSM into 10s fully resolves the issue.

INFO submit response to cloudformation

{
	"Status": "FAILED",
	"Reason": "ValidationException: 1 validation error detected: Value '[/cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydrahydrapublicB75FC865RefEA95D68D, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2orykratoskratosadmin66E5C2F5RefABC2E741, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttsamlfederationNestedStacksamlfederationNestedStackResourceF9BAFFEBOutputscompanyproductjosh22apsoutheast2samlfederationalb8BC22212Ref5884B1DC, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydrahydraadmin44A1DC5ARef25110A0F, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttadmingqlNestedStackadmingqlNestedStackResource7F92880AOutputscompanyproductjosh22apsoutheast2admingqlalbEF458438RefF1172A0D, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryproxyNestedStackproxyNestedStackResource12C3FA79Outputscompanyproductjosh22apsoutheast2oryproxyal9F2CB0FD, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydraadhydraadadmin13715108Ref123ADE30, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttmigrationNestedStackmigrationNestedStackResourceC0800D01Outputscompanyproductjosh22apsoutheast2migrationalb2B1D1A53Ref0B868277, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2orykratoskratospublicBF598F48Ref61F05EEB, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttsamlNestedStacksamlNestedStackResource2B1E690EOutputscompanyproductjosh22apsoutheast2samlalb776F7A60Ref2A2E0028, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttoryNestedStackoryNestedStackResource908720F1Outputscompanyproductjosh22apsoutheast2oryhydraadhydraadpublic8B011CFARef0C11F132, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttloginNestedStackloginNestedStackResource5E1A21F5Outputscompanyproductjosh22apsoutheast2loginalb6D8F1A33Ref5E92FD74, /cdk/exports/companyproductjosh22acc66077BB2/companyproductjosh22apsoutheast2A61AF646apsoutheast2FnGetAttactivedirectoryfederationNestedStackactivedirectoryfederationNestedStackResourceF6043CF6Outputscompanyproductjosh22apsoutheast2activedirectoryfederationalb6E7DB668Ref1F7C174D]' at 'names' failed to satisfy constraint: Member must have length less than or equal to 10\n at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)\n at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)\n at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)",
	"StackId": "arn:aws:cloudformation:ap-southeast-2:028446965111:stack/companyproductjosh22apsoutheast2A61AF646/5ae288d0-a237-11ed-9f7a-06ed84041138",
	"RequestId": "91a5269f-09b3-466f-85e3-84b35afc615d",
	"PhysicalResourceId": "40ad19e5-12d3-4330-bae5-56a5587df2dd",
	"LogicalResourceId": "ExportsWritereuwest25F52B32F9AB3BFFD"
}

[tim-finnigan]

Thanks @joshbalfour for creating the PR, the team will try to review it when time permits.

[colifran]

This is a CloudFormation limitation stemming from custom resource provider responses being limited to 4096 bytes - docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref-responses.html. In the specific use case you've provided you can fix this by shortening the overall stack name by either shortening the stack prefix or by not using nested stacks which is adding to the stack name length. Longer term I'm not sure there is a good alternative here besides maybe introducing a way to control the parameter name length. We can't truncate or provide something like outputPaths since all of the imports are needed in the response back to CloudFormation. This PR adds some documentation explaining this - #30115

[colifran]

Added documentation for this here and opened a new issue to track this for crossRegionReferences as a whole - #30119

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.