EKS Pod Identity - Cannot get credentials

[leroylim20]

Describe the bug

Using Java SDK 1.12.596, the pod with Pod Identity addon is not able to get AWS credentials via SDK. On the same pod, using AWS CLI v1, it can successfully get its credentials.

On AWS CLI v1

root@f6bd48d86-7x4qn:/# aws --version
aws-cli/1.31.5 Python/3.10.12 Linux/5.10.198-187.748.amzn2.x86_64 botocore/1.33.5
root@f6bd48d86-7x4qn:/# aws ec2 describe-instances
{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "<redacted",
                    "InstanceId": "<redacted>",
                    "InstanceType": "c5.large",

.
.
.

Expected Behavior

Successfully runs the AWS command (E.g. EC2 describe-instances) and prints out its output

Current Behavior

kubectl exec pod/f6bd48d86-7x4qn -c test -it -- bash
root@f6bd48d86-7x4qn:/# curl localhost:8080/version
1.12.596
root@test-f6bd48d86-7x4qn:/# curl localhost:8080/AZ
{"timestamp":"2023-12-01T06:44:30.270+00:00","status":500,"error":"Internal Server Error","path":"/AZ"}
root@f6bd48d86-7x4qn:/# exit
exit

Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@b35605f: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4bed011b: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]] with root cause

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@b35605f: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4bed011b: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]
	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:142) ~[aws-java-sdk-core-1.12.596.jar!/:na]

Reproduction Steps

Run a pod with Java AWS SDK that calls AWS API with pod identity configured

    @GetMapping("/AZ")
    String getAZ() {
        String node_name = System.getenv("NODE_NAME");
        final AmazonEC2 ec2 = AmazonEC2ClientBuilder.defaultClient();

        DescribeInstancesRequest request = new DescribeInstancesRequest().withFilters(new Filter("network-interface.addresses.private-dns-name", Collections.singletonList(node_name)));
        DescribeInstancesResult result = ec2.describeInstances(request);

        String az;
        Optional<Instance> instance;

        if (result.getReservations() != null) {
                instance = result.getReservations().stream().flatMap(reservation -> reservation.getInstances().stream()).findFirst();
                if (instance.isPresent()) {
                    az = instance.get().getPlacement().getAvailabilityZone();
                }
                else {
                System.out.printf(
                        "No instances found with %s%n",
                        node_name);
                return "No instance found";
            }
        }
        else {
            return "API Error";
        }

Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      serviceAccountName: test-serviceaccount
      containers:
      - name: test
        image: <redacted>
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Possible Solution

Fix the credential chain for Pod Identity

Additional Information/Context

No response

AWS Java SDK version used

1.12.596

JDK version used

11

Operating System and version

eclipse-temurin:11

[cademarkegard]

I'm experiencing the same issue using AWS SDK version 1.12.619

[cademarkegard]

looks like V1 here: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/ContainerCredentialsProvider.java#L144-L159

Should follow the pattern that is in V2:
https://github.com/aws/aws-sdk-java-v2/blob/master/core/auth/src/main/java/software/amazon/awssdk/auth/credentials/ContainerCredentialsProvider.java#L278-L290

[debora-ito]

@leroylim20 @cademarkegard sorry for the long silence.

Java SDK 1.x does not support EKS Pod Identity.

2.21.30 is the minimum SDK version that supports EKS Pod Identity - see the list of SDKs and their minimum versions in the Amazon EKS User Guide.

[cademarkegard]

hi @debora-ito thanks for the response however it is listed in the CHANGELOG that it is supposed to: https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md#amazon-eks-auth .

Can you please update the documentation if AWS does not have plans on supporting this feature to not mislead others in the future?

[debora-ito]

Can you please update the documentation if AWS does not have plans on supporting this feature to not mislead others in the future?

Yes, I'll reach out to the EKS documentation team and ask to clarify here the Java SDK v1 does not support EKS Pod Identity.

The changelog entry is the service release note that is automatically sent to all the AWS SDKs when a service feature is released that day. Service features usually don't require changes on the SDK side, but in the case of Pod Identities the Container Credential Provider process required some changes, and those changes were implemented in Java SDK v2 only, they won't be in v1.

[cademarkegard]

Thanks so much for the explanation!

[daniel-dios]

Hey it looks like they changed documentation yesterday: <lastBuildDate>Mon, 19 Feb 2024 04:47:18 GMT</lastBuildDate>

I see now EKS is supporting v1: https://docs.aws.amazon.com/pdfs/eks/latest/userguide/eks-ug.pdf#pod-id-minimum-sdk and the error keeps happening.

Wondering if the problem got solved in following versions or what!

cc @rijulGogia

[debora-ito]

Ok, an update:

Java SDK 1.x doesn't support EKS Pod Identity. The documentation change pointed out by @daniel-dios was made by mistake, and I asked the Documentation team to rectify - there's a banner there now saying

An earlier version of the documentation was incorrect. The AWS SDK for Java v1 doesn't support 
EKS Pod Identity.

@cademarkegard after I asked the Doc team to explicitly add the Java v1 to the list of SDKs and highlight that is not supported, they replied saying the list should include only the SDKs that do support, and they'd like to keep it this way. So hopefully the banner will call attention to it, while it's there.

I'll mark this to autoclose soon. Let me know if there's any follow-up questions.

[flozano]

there is no plan to support EKS Pod identity in AWS SDK V1? rewriting entire services is not an option in most cases.

[debora-ito]

That's correct, there's no plans to support EKS Pod Identity in V1. This aligns with the V1 deprecation path announced here, V1 will enter maintenance mode July 31, 2024.

[flozano]

that's unfortunate, one would expect that maintenance mode entering in July and being in March, this would be available (general availability ends on July 30, and this was notified just 2 months ago).

That way we would have a little more time to "port" components, while infrastructure can move forward with deploying EKS Pod identity.

Is there any technical difficulty to implement it externally in a custom credentials provider ? Just asking because I may be tempted to attempt implementing it - unless there's a blocker or high technical toll that prevents it.

[flozano]

Any update on this?

[flozano]

Any technical insight about why this was discarded (technical impossibility, high difficulty?) would be highly appreciated in order to help me deciding if I can implement myself. Does this require any kind of signature that is not supported in AWS SDK v1 or something of the kin?

Edit: Just to confirm, I'm looking at the code and it seems the only change needed would be to also consider AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE aside from AWS_CONTAINER_AUTHORIZATION_TOKEN. Is that correct? Am I missing something?

[flozano]

Just to confirm: for us it's working well with this sample project https://github.com/flozano/aws-eks-identity-pod-v1

Given the minimal extent of the changes, the fact that AWS SDK v1 maintenance mode was just recently announced (at least publicly - I'm aware it was announced inside AWS much earlier) and that maintenance mode will not start until July (4 months away), I'm not sure how AWS justifies not supporting this feature.

[flozano]

I'm wondering what's reasonable to expect from SDK v1 updates, given that in:

https://aws.amazon.com/es/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/#:~:text=Between%20July%2031,%202024%20and,or%20changes%20to%20existing%20services

we are still in General Availability period, and

During this phase, the SDK is fully supported. AWS will provide regular SDK releases that include support for new services, API updates for existing services, as well as bug and security fixes.

just saying.