(aws-eks): Unable to deploy cluster in regions that are not enabled by default

[civilizeddev]

In AWS Bahrain region (me-south-1), establishing an EKS cluster with cdk has been failed,

with this error message:

"AWS was not able to validate the provided access credentials"

While it succeeded in AWS Seoul region (ap-northeast-2)

It seems to be related to the STS endpoint problem.

Reproduction Steps

My Stack contains:

import * as cdk from '@aws-cdk/core'
import * as eks from '@aws-cdk/aws-eks'

export class EksStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props)
    new eks.Cluster(this, 'cluster', {})
  }
}

// ...

const app = new cdk.App()
new EksStack(app, 'eks', { env: { region: 'me-south-1' } })
app.synth()

When I run it:

$ cdk deploy eks

It is aborted with an error like:

ERROR: AWS was not able to validate the provided access credentials

What did you expect to happen?

The stack should be successfully deployed.

What actually happened?

Failed to create resource. The security token included in the request is invalid

Logs: /aws/lambda/eks-awscdkawseksClusterReso-OnEventHandler42BEBAE0-4QPUO1YX3M0U

Failed to delete resource. The security token included in the request is invalid

Logs: /aws/lambda/eks-awscdkawseksClusterRe-IsCompleteHandler7073F4D-ER83I62J68AJ

The stack deployment was aborted at this stage:

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster.ts

Lines 1007 to 1030 in 77ce45d

	const resource = this._clusterResource = new ClusterResource(this, 'Resource', {
	name: this.physicalName,
	environment: props.clusterHandlerEnvironment,
	roleArn: this.role.roleArn,
	version: props.version.version,
	resourcesVpcConfig: {
	securityGroupIds: [securityGroup.securityGroupId],
	subnetIds,
	},
	...(props.secretsEncryptionKey ? {
	encryptionConfig: [{
	provider: {
	keyArn: props.secretsEncryptionKey.keyArn,
	},
	resources: ['secrets'],
	}],
	} : {} ),
	endpointPrivateAccess: this.endpointAccess._config.privateAccess,
	endpointPublicAccess: this.endpointAccess._config.publicAccess,
	publicAccessCidrs: this.endpointAccess._config.publicCidrs,
	secretsEncryptionKey: props.secretsEncryptionKey,
	vpc: this.vpc,
	subnets: placeClusterHandlerInVpc ? privateSubents : undefined,
	});

Environment

• CDK CLI Version : 1.94.1
• Framework Version:
• Node.js Version: 14.16.0
• OS : macOS BigSur 11.2.3(20D91)
• Language (Version): TypeScript 4.2.3

Other

• Managing AWS STS in an AWS Region
• Why did I receive the IAM error "AWS was not able to validate the provided access credentials" in some AWS Regions?

---

This is 🐛 Bug Report

[civilizeddev]

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource-handler/index.ts

Line 31 in 77ce45d

eks = new aws.EKS({ credentials: creds });

[civilizeddev]

In some AWS regions, such as ap-east-1 and me-south-1,

aws-sdk and aws-cli behave differently when calling STS endpoints.

To fix this to do the same in all the regions,

lambda functions used in cluster-resource-provider in cdk should have an environment variable:

• AWS_STS_REGIONAL_ENDPOINTS: regional

aws/aws-cli#4370 (comment)

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource-provider.ts

Lines 61 to 70 in 77ce45d

	const onEvent = new lambda.Function(this, 'OnEventHandler', {
	code: lambda.Code.fromAsset(HANDLER_DIR),
	description: 'onEvent handler for EKS cluster resource provider',
	runtime: HANDLER_RUNTIME,
	environment: props.environment,
	handler: 'index.onEvent',
	timeout: Duration.minutes(1),
	vpc: props.subnets ? props.vpc : undefined,
	vpcSubnets: props.subnets ? { subnets: props.subnets } : undefined,
	});

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource-provider.ts

Lines 71 to 80 in 77ce45d

	const isComplete = new lambda.Function(this, 'IsCompleteHandler', {
	code: lambda.Code.fromAsset(HANDLER_DIR),
	description: 'isComplete handler for EKS cluster resource provider',
	runtime: HANDLER_RUNTIME,
	handler: 'index.isComplete',
	timeout: Duration.minutes(1),
	vpc: props.subnets ? props.vpc : undefined,
	vpcSubnets: props.subnets ? { subnets: props.subnets } : undefined,
	});

To be:

new lambda.Function(this, 'XXXHandler', {
  ...,
  environment: {
    ...props.environment,
    AWS_STS_REGIONAL_ENDPOINTS: 'regional'
  }
}

[iliapolo]

@civilizeddev Thanks for reporting this.

You can workaround this issue for now by applying an aspect that injects that environment variable to all lambda functions associated with the cluster:

const cluster = new eks.Cluster(...)
cdk.Aspects.of(Stack.of(cluster)).add({
  visit: (node: cdk.IConstruct) => {
    if (node instanceof lambda.CfnFunction) {
      node.addPropertyOverride('Environment.Variables.AWS_STS_REGIONAL_ENDPOINTS', 'regional')
    }
  }
})

Another option, though less recommended, is enabling global STS endpoints for all regions in your account.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.