aws-cdk: assumed roles using source_profile from SSO sources cannot get credentials #19897

elliotsegler · 2022-04-13T10:13:28Z

Describe the bug

When trying to acquire credentials using a assumed role source_profile connected to an SSO profile, I receive the following error

[Error at /CdkStack] Need to perform AWS calls for account 123456789012, but no credentials have been configured

Expected Behavior

Credentials to be loaded correctly, and ProcessCredentialsProviderFailure not to be called. ProcessCredentialsProvider should not be used in this case.

Current Behavior

Trace logs:

[2022-04-13 17:52:58] user@PC MSYS /c/Dev/ws/CDKTestProject/cdk
$ npx cdk synth -v --profile my-assumed-sso-role
CDK toolkit version: 2.20.0 (build 738ef49)
Command line arguments: {
  _: [ 'synth' ],
  v: 1,
  verbose: 1,
  profile: 'my-assumed-sso-role',
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  validation: true,
  quiet: false,
  q: false,
  '$0': 'node_modules\\aws-cdk\\bin\\cdk'
}
cdk.json: {
  "app": "npx ts-node --prefer-ts-exts bin/app.ts",
  "watch": {
    "include": [
      "**"
    ],
    "exclude": [
      "README.md",
      "cdk*.json",
      "**/*.d.ts",
      "**/*.js",
      "tsconfig.json",
      "package*.json",
      "yarn.lock",
      "node_modules",
      "test"
    ]
  },
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
    "@aws-cdk/core:stackRelativeExports": true,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
    "@aws-cdk/core:target-partitions": [
      "aws",
      "aws-cn"
    ]
  }
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'npx ts-node --prefer-ts-exts bin/app.ts',
  watch: {
    include: [ '**' ],
    exclude: [
      'README.md',
      'cdk*.json',
      '**/*.d.ts',
      '**/*.js',
      'tsconfig.json',
      'package*.json',
      'yarn.lock',
      'node_modules',
      'test'
    ]
  },
  context: {
    '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
    '@aws-cdk/core:stackRelativeExports': true,
    '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
    '@aws-cdk/aws-lambda:recognizeVersionProps': true,
    '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true,
    '@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver': true,
    '@aws-cdk/aws-ec2:uniqueImdsv2TemplateName': true,
    '@aws-cdk/core:target-partitions': [ 'aws', 'aws-cn' ]
  },
  debug: false,
  assetMetadata: true,
  profile: 'my-assumed-sso-role',
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '*' ],
  lookups: true
}
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to ap-southeast-2
Resolving default credentials
Unable to determine the default AWS account: ProcessCredentialsProviderFailure: Profile my-assumed-sso-role did not include credential process
    at ProcessCredentials2.load (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials\process_credentials.js:102:11)
    at ProcessCredentials2.coalesceRefresh (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials.js:205:12)
    at ProcessCredentials2.refresh (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials\process_credentials.js:163:10)
    at ProcessCredentials2.get2 [as get] (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials.js:122:12)
    at resolveNext2 (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials\credential_provider_chain.js:125:17)
    at C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials\credential_provider_chain.js:126:13
    at C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials.js:124:23
    at C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-sdk\lib\credentials.js:212:15
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 'ProcessCredentialsProviderFailure',
  time: 2022-04-13T09:53:07.307Z
}
context: {
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
  '@aws-cdk/core:stackRelativeExports': true,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
  '@aws-cdk/aws-lambda:recognizeVersionProps': true,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true,
  '@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver': true,
  '@aws-cdk/aws-ec2:uniqueImdsv2TemplateName': true,
  '@aws-cdk/core:target-partitions': [ 'aws', 'aws-cn' ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'ap-southeast-2',
  CDK_CONTEXT_JSON: '{"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId":true,"@aws-cdk/core:stackRelativeExports":true,"@aws-cdk/aws-rds:lowercaseDbIdentifier":true,"@aws-cdk/aws-lambda:recognizeVersionProps":true,"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021":true,"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver":true,"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName":true,"@aws-cdk/core:target-partitions":["aws","aws-cn"],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true,"aws:cdk:version-reporting":true,"aws:cdk:bundling-stacks":["*"]}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '17.0.0',
  CDK_CLI_VERSION: '2.20.0'
}
Notices refreshed
Some context information is missing. Fetching...
Setting "vpc-provider:account=123456789012:filter.vpc-id=vpc-00000000000000000:region=ap-southeast-2:returnAsymmetricSubnets=true" context to {"$providerError":"Need to perform AWS calls for account 123456789012, but no credentials have been configured","$dontSaveContext":true}
Setting "CDK_DEFAULT_REGION" environment variable to ap-southeast-2
context: {
  'vpc-provider:account=123456789012:filter.vpc-id=vpc-00000000000000000:region=ap-southeast-2:returnAsymmetricSubnets=true': {
    '$providerError': 'Need to perform AWS calls for account 123456789012, but no credentials have been configured',
    '$dontSaveContext': true
  },
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
  '@aws-cdk/core:stackRelativeExports': true,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
  '@aws-cdk/aws-lambda:recognizeVersionProps': true,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true,
  '@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver': true,
  '@aws-cdk/aws-ec2:uniqueImdsv2TemplateName': true,
  '@aws-cdk/core:target-partitions': [ 'aws', 'aws-cn' ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'ap-southeast-2',
  CDK_CONTEXT_JSON: '{"vpc-provider:account=123456789012:filter.vpc-id=vpc-00000000000000000:region=ap-southeast-2:returnAsymmetricSubnets=true":{"$providerError":"Need to perform AWS calls for account 123456789012, but no credentials have been configured","$dontSaveContext":true},"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId":true,"@aws-cdk/core:stackRelativeExports":true,"@aws-cdk/aws-rds:lowercaseDbIdentifier":true,"@aws-cdk/aws-lambda:recognizeVersionProps":true,"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021":true,"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver":true,"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName":true,"@aws-cdk/core:target-partitions":["aws","aws-cn"],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true,"aws:cdk:version-reporting":true,"aws:cdk:bundling-stacks":["*"]}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '17.0.0',
  CDK_CLI_VERSION: '2.20.0'
}
Not making progress trying to resolve environmental context. Giving up.
[Error at /CdkStack] Need to perform AWS calls for account 123456789012, but no credentials have been configured
  Annotations.addMessage (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk-lib\core\lib\annotations.ts:99:25)
  Annotations.addError (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk-lib\core\lib\annotations.ts:58:10)
  Function.getValue (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk-lib\core\lib\context-provider.ts:111:31)
  Function.fromLookup (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk-lib\aws-ec2\lib\vpc.ts:1161:66)
  new CdkStack (C:\Dev\ws\CDKTestProject\cdk\lib\cdk-stack.ts:20:33)
  Object.<anonymous> (C:\Dev\ws\CDKTestProject\cdk\bin\app.ts:8:1)
  Module._compile (node:internal/modules/cjs/loader:1103:14)
  Module.m._compile (C:\Dev\ws\CDKTestProject\cdk\node_modules\ts-node\src\index.ts:1056:23)
  Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
  Object.require.extensions.<computed> [as .ts] (C:\Dev\ws\CDKTestProject\cdk\node_modules\ts-node\src\index.ts:1059:12)
  Module.load (node:internal/modules/cjs/loader:981:32)
  Function.Module._load (node:internal/modules/cjs/loader:822:12)
  Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
  main (C:\Dev\ws\CDKTestProject\cdk\node_modules\ts-node\src\bin.ts:198:14)
  Object.<anonymous> (C:\Dev\ws\CDKTestProject\cdk\node_modules\ts-node\src\bin.ts:288:3)
  Module._compile (node:internal/modules/cjs/loader:1103:14)
  Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
  Module.load (node:internal/modules/cjs/loader:981:32)
  Function.Module._load (node:internal/modules/cjs/loader:822:12)
  Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
  node:internal/main/run_main_module:17:47

Found errors
Error: Found errors
    at StackCollection.processMetadataMessages (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk\lib\api\cxapp\cloud-assembly.ts:274:13)
    at CdkToolkit.validateStacks (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk\lib\cdk-toolkit.ts:644:12)
    at CdkToolkit.selectStacksForDiff (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk\lib\cdk-toolkit.ts:623:10)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at CdkToolkit.synth (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk\lib\cdk-toolkit.ts:507:20)
    at initCommandLine (C:\Dev\ws\CDKTestProject\cdk\node_modules\aws-cdk\lib\cli.ts:342:12)

Reproduction Steps

~/.aws/config contains SSO profile and an assumed role profile

$ cat ~/.aws/config
[profile default]
region = ap-southeast-2

[profile my-sso-role]
sso_start_url = https://XXXXXXXX.awsapps.com/start
sso_region = ap-southeast-2
sso_account_id = 123456789012
sso_role_name = 123456789012-DevOps
region = ap-southeast-2

[profile my-assumed-sso-role]
source_profile = my-sso-role
role_arn = arn:aws:iam::123456789012:role/Role-DevOps-IP-Restricted
region = ap-southeast-2

~/.aws/credentials is currently empty, but I've tried with a default profile and it doesn't make a difference

Running npx cdk synth --profile my-assumed-sso-role causes the error.

Running aws sts get-caller-identity --profile my-assumed-sso-role shows a valid session.

Possible Solution

Possibly bump the bundled aws-sdk version

Additional Information/Context

No response

CDK CLI Version

2.20.0 (build 738ef49)

Framework Version

No response

Node.js Version

v16.14.2

OS

Windows

Language

Typescript

Language Version

No response

Other information

I suspect that it's actually an upstream issue with aws-sdk related to source_profile which was recently fixed, such as aws/aws-sdk-js-v3#2221

The text was updated successfully, but these errors were encountered:

elliotsegler · 2022-04-14T01:11:21Z

I might be wrong about it being an upstream issue. I've tested swapping the source profile (using SSO) with static credentials, and that appears to work ok, so it might be an issue with the credentials provider chain in packages/aws-cdk/lib/api/aws-auth/awscli-compatible.ts.

AWS Config that worked:

~/.aws/config

[profile default]
region = ap-southeast-2

[profile my-assumed-sso-role]
source_profile = 123456789012-DevOps
role_arn = arn:aws:iam::123456789012:role/Role-DevOps-IP-Restricted
region = ap-southeast-2

~/.aws/credentials

[default]
region=ap-southeast-2

[my-assumed-sso-role]
source_profile = 123456789012-DevOps
role_arn = arn:aws:iam::123456789012:role/Role-DevOps-IP-Restricted
region = ap-southeast-2

[123456789012-DevOps]
aws_access_key_id=ASIAREDACTED
aws_secret_access_key=REDACTED
aws_session_token=REDACTED

skinny85 · 2022-04-14T02:51:00Z

Thanks for opening the issue @elliotsegler!

@comcalvi you might want to take a look at this one.

elliotsegler · 2022-04-19T01:28:17Z

I've been tinkering with this and aws-sdk and I think the issue is related to the AWS.SharedIniFileCredentials and by extension CDK's PatchedSharedIniFileCredentials.

In this case, my-assumed-sso-role is being eaten by the PatchedSharedIniFileCredentials implementation. When that fetches the source credentials in sourceProfileCredentials() we're specifying that the source profile is also fetched using AWS.SharedIniFileCredentials and not AWS.SsoCredentials if if needed to be SSO typed.

I think there's actually two issues here:

The issue I have described with SSO credentials
Because source_profile leads to the underlying AWS.SharedIniFileCredentials implementation, any profile loaded there won't have issues fixed where they have been patched in the PatchedSharedIniFileCredentials class.

This particular issue was fixed upstream, but in the v3 SDK with some resolver logic (see aws-sdk-js-v3:packages/credential-provider-ini/src)

comcalvi · 2022-04-19T21:55:47Z

@elliotsegler I haven't been able to reproduce this yet. This is my ~/.aws/config.

$ cat ~/.aws/config
[default]
region = us-east-1

[profile my-sso-profile]
sso_start_url = https://d-abc123def4.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789012
sso_role_name = MySsoRoleName
region = us-east-1

[profile my-assumed-sso-role]
source_profile = my-sso-profile
role_arn = arn:aws:iam::123456789012:role/aws-reserved/sso.amazonaws.com/AWSSSOROLE
region = us-east-1

~/.aws/credentials is an empty file. Running aws sso login --profile=my-sso-profile and then running npx cdk synth --profile my-assumed-sso-role successfully synths the stack. Can you ensure that the version of aws-cdk-lib is the same as aws-cdk?

elliotsegler · 2022-04-20T04:55:24Z

Hey @comcalvi

Checking my copy of the CDK, yes - the versions are the same.

[2022-04-20 12:52:23] user@PC MSYS /c/Dev/ws/CDKTestProject/cdk (master)
$ npm list aws-cdk-lib aws-cdk
cdk-test-project@0.1.0 C:\Dev\ws\CDKTestProject\cdk
├── aws-cdk-lib@2.20.0
└── aws-cdk@2.20.0

I think the issue with your reproduction is an issue with your ~/.aws/config. The role_arn in the my-assumed-sso-role needs to be a role that's trusted an existing sso role, not the specific SSO role itself. To reproduce you'll need two roles.

arn:aws:iam::123456789012:role/aws-reserved/sso.amazonaws.com/MySsoRoleName

Needs to have an assume role policy that trusts AWS SSO's SAML provider
Needs to have permissions that only allow it to AssumeRole to the role_arn in the my-assumed-sso-role

my-assumed-sso-role

Create an IAM role in the account 123456789012 that has an assume role policy that trusts arn:aws:iam::123456789012:role/aws-reserved/sso.amazonaws.com/MySsoRoleName
Allow permissions to do normal things
Get the arn of this role and put it in the role_arn field in your profile for my-assumed-sso-role

elliotsegler · 2022-04-20T04:58:25Z

I'll also suggest that my ~/.aws/credentials file is definitely not empty. I think that matters, because of the way the AWS.SharedIniFileCredentials and by extension PatchedSharedIniFileCredentials processes that file.

comcalvi · 2022-04-21T00:36:26Z

Thanks for the detailed explanation @elliotsegler. I'm still working on a reproduction, but with an empty credentials file, I can use a role that trusts the sso role, as you described, and successfully synth a stack. I'll look more into this later.

elliotsegler · 2022-04-22T05:23:12Z

but with an empty credentials file, I can use a role that trusts the sso role, as you described, and successfully synth a stack.

@comcalvi: Are you seeing issues resolving credentials while synthing? Being able to synth a stack is not what I'd consider success criteria. It's possible for the credentials process to fail and for the stack to still synth. This is the same for any cdk command like cdk ls or cdk synth.

In the case I've got, I'm using a VPC lookup. So on a synth with an empty context it would fail. If I manually set the context, or I use a non-sso profile/creds to fetch and set the context, every subsequent synth works even if it can't fetch credentials.

Are you able to cdk deploy using the assumed role?

comcalvi · 2022-05-06T21:47:00Z

you're right, cdk deploy fails, even though cdk synth succeeds silently. cdk deploy fails with

Unable to resolve AWS account to use. It must be either configured when you define your CDK Stack, or through the environment

for both the SSO role and the role that trusts the existing SSO role.

This is definitely broken, I'll be working on a fix for this.

comcalvi · 2022-05-13T20:28:58Z

turns out when using source profiles, the CDK will always try to create a SharedIniFileCredentials, which SSO credentials are not. Adding SSO credentials to the source profile loading logic resolves this.

SSO profiles can now be used as source profiles. Fixes #19897. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions · 2022-05-14T07:14:49Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

SSO profiles can now be used as source profiles. Fixes aws#19897. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

elliotsegler added bug needs-triage labels Apr 13, 2022

github-actions bot added the @aws-cdk/aws-sso label Apr 13, 2022

github-actions bot assigned skinny85 Apr 13, 2022

skinny85 assigned comcalvi and unassigned skinny85 Apr 14, 2022

comcalvi added the investigating label Apr 19, 2022

peterwoodworth removed the needs-triage label Apr 19, 2022

comcalvi added the needs-reproduction label Apr 19, 2022

comcalvi added p1 and removed investigating needs-reproduction labels May 6, 2022

comcalvi mentioned this issue May 13, 2022

fix(CLI): allow SSO profiles to be used as source profiles #20340

Merged

mergify bot closed this as completed in #20340 May 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-cdk: assumed roles using source_profile from SSO sources cannot get credentials #19897

aws-cdk: assumed roles using source_profile from SSO sources cannot get credentials #19897

elliotsegler commented Apr 13, 2022

elliotsegler commented Apr 14, 2022

skinny85 commented Apr 14, 2022

elliotsegler commented Apr 19, 2022

comcalvi commented Apr 19, 2022 •

edited

Loading

elliotsegler commented Apr 20, 2022 •

edited

Loading

elliotsegler commented Apr 20, 2022

comcalvi commented Apr 21, 2022 •

edited

Loading

elliotsegler commented Apr 22, 2022

comcalvi commented May 6, 2022

comcalvi commented May 13, 2022

github-actions bot commented May 14, 2022

aws-cdk: assumed roles using source_profile from SSO sources cannot get credentials #19897

aws-cdk: assumed roles using source_profile from SSO sources cannot get credentials #19897

Comments

elliotsegler commented Apr 13, 2022

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

elliotsegler commented Apr 14, 2022

skinny85 commented Apr 14, 2022

elliotsegler commented Apr 19, 2022

comcalvi commented Apr 19, 2022 • edited Loading

elliotsegler commented Apr 20, 2022 • edited Loading

elliotsegler commented Apr 20, 2022

comcalvi commented Apr 21, 2022 • edited Loading

elliotsegler commented Apr 22, 2022

comcalvi commented May 6, 2022

comcalvi commented May 13, 2022

github-actions bot commented May 14, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

comcalvi commented Apr 19, 2022 •

edited

Loading

elliotsegler commented Apr 20, 2022 •

edited

Loading

comcalvi commented Apr 21, 2022 •

edited

Loading