winglibs/sagemaker doesn't work with aws sso login

[ekeren]

I tried this:

As reported by @bencaldwell.

When trying to use winglibs/sagemaker with sso based login

Notice that we also used AWS_PROFILE and AWS_REGION

let sm = new sagemaker.Endpoint("some endpoint", "some inference");
let handler = inflight () => {
  let res = sm.invoke({
    body: "do AI stuff"
    },
    ContentType: "application/json"
  );
  log(res.Body);
};
new cloud.Function(handler, env: {
  "AWS_PROFILE":"some-profile",
  "AWS_REGION": "some-region"
});

This happened:

could not load credentials from any providers

I expected this:

To invoke the model

Is there a workaround?

No response

Anything else?

It seems that the auth in winglibs/sagemaker and winglibs/bedrock, doesn't work with short lived TOKEN

There are some reference to this in the following tickets (although I am not sure if this is related)

• SSO Credentials Provider fails to load AWS credentials if the AWS IAM Identity Center access token associated with an SSO session has expired or is about to expire aws/aws-sdk-js#4441
• cli: CDK does not seem to be compatible with config generated from aws configure sso aws/aws-cdk#27265 (comment)

Wing Version

0.57.22

Node.js Version

No response

Platform(s)

No response

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.

[skorfmann]

so to understand what's the goal here: You wanna run this locally in the simulator (or as a test) with an active AWS SSO session. And the compiled wing handlers are not picking up the credentials from the SSO session, right?

Don't know of the top of my head, but happy to look into this. Skimming through the linked issues, these suggest that it might be due to some incompatible fields in the aws config file. Another thought I just had was, that maybe esbuild bundling might be interfering here.

Can look deeper into this later today / tomorrow. Is there a full code example somewhere?

[ekeren]

so to understand what's the goal here: You wanna run this locally in the simulator (or as a test) with an active AWS SSO session. And the compiled wing handlers are not picking up the credentials from the SSO session, right?

Yes the goal is to run this in the simulator (not test, test should use a mock) on interactive development with an active SSO session.

And the compiled wing handlers are not picking up the credentials from the SSO session, right?
Yes

is there a full code example

It is basically https://github.com/winglang/winglibs/tree/main/sagemaker

Can look deeper into this later today / tomorrow.

It might be the @tsuf239 will work on it, I am not sure.

Lets see what @ainvoner / @staycoolcall911 have to say about this.

[skorfmann]

Just looked into this a bit. The bundled code contains the relevant credential providers. To confirm, I changed the function in this example to https://github.com/winglang/winglibs/blob/main/sagemaker/sagemaker.test.w

new cloud.Function(invokeModel,
  env: {
    "AWS_PROFILE":"wing-sebastian"
});

where the ~/.aws/config file contains the following and the ~/.aws/credentials file is empty:

[profile wing-sebastian]
sso_session = wing
sso_account_id = <account_id>
sso_role_name = <role>
region = <region>
output = json

[sso-session wing]
sso_start_url = <sso_url>
sso_region = <sso_region>
sso_registration_scopes = sso:account:access

after doing a sign in via AWS_PROFILE=wing-sebastian aws sso login I was able to invoke the function in the wing simulator. Well, I got error: ValidationError: Endpoint Example-endopint of account <account_id> not found., but that's not related to credentials.

My guess would be, that the SSO setup is either outdated (try upgrading the AWS CLI and see if a login works) or perhaps the SSO session is broken for some reason. A quick check could be if AWS_PROFILE=my-profile aws sts get-caller-identity returns an active session.

If that's all not helping, we'd need more information about the environment. Also, I'd probably try to write a minimal nodejs script which does the similar thing without wing to see if that's working.

[skorfmann]

Besides from the things mentioned above: Perhaps it would be worthwhile if the simulator would pass through relevant AWS_* environment variables. Essentially a small change to the sandbox init.

wing/libs/wingsdk/src/shared/sandbox.ts

Line 39 in c4e9312

env: this.options.env,

[bencaldwell]

I have the aws sso config setup in the same way as @skorfmann. The profile works correctly when I call aws sts get-caller-identity.

It works when I run it as a test, but not when I invoke in the simulator. To elaborate:

Works fine:
I have a main.test.w test based on the sagemaker example.
When I run wing test main.test.w the test passes.

Doesn't work:
With the same config when I run wing run main.w and try to invoke from the simulator I get response { "success": false, "error": "Could not load credentials from any providers" }

The code for main.w is below.

// main.w

bring ex;
bring cloud;
bring util; // Import the base64 package
bring http;
bring sagemaker;

let sm = new sagemaker.Endpoint("asdf", "qwerty");

let handler = inflight () => {
  let res = sm.invoke(
    {"img_base64": "asdf"},
    ContentType: "application/json"
  );
  log(res.Body);
};

new cloud.Function(handler, env: {
  AWS_PROFILE: "my-profile",
  AWS_REGION: "ap-southeast-2"
});

aws --version output:
aws-cli/2.15.21 Python/3.11.6 Linux/5.15.133.1-microsoft-standard-WSL2 exe/x86_64.ubuntu.22 prompt/off

Installed wing modules:
@winglibs/sagemaker@0.0.2
winglang@0.58.10

[skorfmann]

hm, interesting.

aws-cli/2.15.21 Python/3.11.6 Linux/5.15.133.1-microsoft-standard-WSL2 exe/x86_64.ubuntu.22 prompt/off

Perhaps an OS issue, since I'm on macOS. What's your exact setup, we could try to reproduce it then more easily.

[bencaldwell]

Below are some details. Let me know if you need anything else?

$wsl --version
WSL version: 2.0.9.0
Kernel version: 5.15.133.1-1
WSLg version: 1.0.59
MSRDC version: 1.2.4677
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19045.4046

$node --version
v20.11.0

$npm list
winglang-webapp@ /home/benc/projects/winglang-webapp
├── @opentelemetry/api@1.7.0 extraneous
├── @winglibs/sagemaker@0.0.2
├── google-p12-pem@4.0.1 extraneous
├── node-forge@1.3.1 extraneous
└── winglang@0.58.10

[skorfmann]

Thanks, I think this should be enough to reproduce.

[polamoros]

I'm getting the same error on my windows machine after configuring the sso:

bring sagemaker;
bring cloud;

let sm = new sagemaker.Endpoint("my-endpoint-name", "my-inference-name");

let invokeModel = inflight () => {
  let res = sm.invoke({
    inputs: "do AI stuff"
    },
    ContentType: "application/json"
  );
  log(res.Body);
};

new cloud.Function(invokeModel, env: {
  "AWS_PROFILE":"my-profile",
});

Error:

Could not load credentials from any providers

[skorfmann]

@polamoros @marciocadev thanks for taking the time to reproduce this issue. Looks like this is particularily related to the standard AWS SSO setup on windows. @marciocadev was able to get a valid session (see here via https://www.leapp.cloud/

A workaround could be, to export the temporary SSO credentials in the environment before starting the wing console, e.g with this command

eval $(aws configure export-credentials --format env)

and then explicitly pass along the following variables in the function definition.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

Implementing this issue #5738 would streamline this even further.

For an actual fix, we'll need to dive deeper into the mechanics of how wing and the js aws sdk plays together on Windows in the context of SSO.

[ekeren]

@bencaldwell , can you check the following workaround:
running it with:

% $(aws configure export-credentials --format env) && wing run

And making sure we pass:

AWS_ACCESS_KEY_ID: util.env("AWS_ACCESS_KEY_ID");
AWS_SECRET_ACCESS_KEY:util.env("AWS_SECRET_ACCESS_KEY");
AWS_SESSION_TOKEN: util.env("AWS_SESSION_TOKEN");

to the cloud.Function

Here is the complete code:

bring ex;
bring cloud;
bring util; // Import the base64 package
bring http;
bring sagemaker;

let sm = new sagemaker.Endpoint("asdf", "qwerty");

let handler = inflight () => {
  let res = sm.invoke(
    {"img_base64": "asdf"},
    ContentType: "application/json"
  );
  log(res.Body);
};

new cloud.Function(handler, env: {
  AWS_PROFILE: "my-profile",
  AWS_REGION: "ap-southeast-2",
  AWS_ACCESS_KEY_ID: util.env("AWS_ACCESS_KEY_ID");
  AWS_SECRET_ACCESS_KEY:util.env("AWS_SECRET_ACCESS_KEY");
  AWS_SESSION_TOKEN: util.env("AWS_SESSION_TOKEN");
});

@skorfmann, FYI

[skorfmann]

AWS_PROFILE: "my-profile", shouldn't be necessary, but also not standing in the way. I believe the credentials have higher priority

[bencaldwell]

Thanks @ekeren and @skorfmann that works!
Please note that I have to delete AWS_PROFILE: "my-profile". If I leave that there it breaks again.

[ekeren]

@tsuf239 FYI