Compatibility with Pod Identity? #147

michal-rybinski · 2024-01-04T11:11:16Z

Is this mechanism also compatible with passing credentials through Pod Identity?

I am testing switching from IRSA to Pod Identity in a K8s cluster and pods which have IRSA configured on them can successfully connect to MSK using this module, whereas each time I try to do it via pod with Pod Identity role, it fails, although I can connect to other services allowed by the IAM role so the mechanism itself works fine.

Is there some specific configuration required for this to work here?

I am a bit at a loss if this is this library problem or kafka's client, but I tried using the newest 3.6.1 from here
https://archive.apache.org/dist/kafka/3.6.1/ alongside with version 2.0.2 of this library and it still throws same errors:

WARN Exception loading credentials. Retry Attempts: 0 (software.amazon.msk.auth.iam.internals.MSKCredentialProvider)
aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@36877a4d: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@3bdccd22: Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@6f468d40: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]]
	at aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:142)
	at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:158)
	at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getCredentials(MSKCredentialProvider.java:145)
	at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:100)
	at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
	at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139)
	at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:534)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:433)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:332)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:273)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1381)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312)
	at java.base/java.lang.Thread.run(Thread.java:829)

AWS Docs are saying here https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html:

"EKS Pod Identities have been added to the Container credential provider which is searched in a step in the default credential chain. If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload."

I am wondering if the problem is in:

library itself
kafka client
client properties (do they need to be adjusted?)

Would appreciate any hints here.

Thanks

sgrabbit · 2024-02-19T21:42:28Z

Hi. Did you find any fix for this?

michal-rybinski · 2024-02-21T11:24:34Z

nope, but was not looking too hard really

ilyas1uphealth · 2024-03-12T00:52:20Z

The issue appears to be with the version of the sdk used in this project -- v1 is not supported
https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html
aws/aws-sdk-java#3062 (comment)

I'm not very familiar with this space, but based on this commit it looks like the project uses v1.12.638 on the latest release
ee55c0f

It looks like to support PodIdentity, this project would need to update that 2.21.30 however I have no idea what that entails.

sidyag · 2024-03-20T12:39:47Z

Can you try this with the PR: #157

This updates all the internal usages of Java sdk v1 to v2.

rajarshp · 2024-04-01T14:14:38Z

Hello,

can this be fixed and release asap ? this is impacting our implementation and we are stuck because of this issue and we are eagerly looking for a solution on this

michal-rybinski · 2024-04-02T08:31:19Z

Can you try this with the PR: #157

This updates all the internal usages of Java sdk v1 to v2.

Hi @sidyag , sorry for long radio silence,

I'll try to see if I can test it soon, although have not done it before without a release I can point my terraform automation at, so will have to research how to do it.

michal-rybinski · 2024-04-04T08:41:38Z

I've noticed that it was merged and see there is version 2.0.3 available now with "Upgrade AWS SKD version" comment (probably should be SDK?) so I gave it a spin today but unfortunately still see the error:

WARN Exception loading credentials. Retry Attempts: 0 (software.amazon.msk.auth.iam.internals.MSKCredentialProvider)
aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@17ead63f: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@f94bdf7: Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@6815eca9: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]]
	at aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:142)
	at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:158)
	at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getCredentials(MSKCredentialProvider.java:145)
	at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:100)
	at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
	at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139)
	at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:534)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:433)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:332)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:273)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1381)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312)
	at java.base/java.lang.Thread.run(Thread.java:829)

although I am definitely using the new library

# ls ../libs/aws*
../libs/aws-msk-iam-auth-2.0.3-all.jar

Do I need to change client.properties in any way (or something else) to use it?

michal-rybinski · 2024-04-04T08:47:24Z

although.... I just noticed that the 2.0.3 release date was 17th of Jan.... :/

michal-rybinski · 2024-04-04T09:09:35Z

ok, got gradle set up and tried to build the newest version cloned but regardless of version used (8.7 or 7.6.4) it fails:

8.7

# gradle clean build
Starting a Gradle Daemon (subsequent builds will be faster)

FAILURE: Build failed with an exception.

* What went wrong:
org/gradle/api/plugins/MavenPlugin
> org.gradle.api.plugins.MavenPlugin

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 19s

7.6.4

# gradle clean build

Welcome to Gradle 7.6.4!

Here are the highlights of this release:
 - Added support for Java 19.
 - Introduced `--rerun` flag for individual task rerun.
 - Improved dependency block for test suites to be strongly typed.
 - Added a pluggable system for Java toolchains provisioning.

For more details see https://docs.gradle.org/7.6.4/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :javadoc FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':javadoc'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/aws-msk-iam-auth/build/tmp/javadoc/javadoc.options'

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6.4/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 36s

sidyag · 2024-04-04T09:12:51Z

Not sure about build fail, as it is passing for me. I am working on running tests, and release.

michal-rybinski · 2024-04-04T09:27:04Z

ok, got it working (installing java-devel helps....)
and the jar generated with gradle clean shadowJar works fine now! 🥇

rajarshp · 2024-04-08T03:32:31Z

Hello,
when can we expect this fix to be released?

sidyag · 2024-04-08T10:04:41Z

The PRs have been released as part of 2.1.0 Could you try and see if that resolves the issue.

michal-rybinski · 2024-04-08T17:18:30Z

Yes, I can confirm it works fine with 2.1.0!

Thank you, closing :)

github-actions · 2024-04-08T17:18:45Z

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

michal-rybinski closed this as completed Apr 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with Pod Identity? #147

Compatibility with Pod Identity? #147

michal-rybinski commented Jan 4, 2024

sgrabbit commented Feb 19, 2024

michal-rybinski commented Feb 21, 2024

ilyas1uphealth commented Mar 12, 2024 •

edited

sidyag commented Mar 20, 2024

rajarshp commented Apr 1, 2024 •

edited

michal-rybinski commented Apr 2, 2024 •

edited

michal-rybinski commented Apr 4, 2024 •

edited

michal-rybinski commented Apr 4, 2024

michal-rybinski commented Apr 4, 2024

sidyag commented Apr 4, 2024

michal-rybinski commented Apr 4, 2024

rajarshp commented Apr 8, 2024

sidyag commented Apr 8, 2024

michal-rybinski commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

Compatibility with Pod Identity? #147

Compatibility with Pod Identity? #147

Comments

michal-rybinski commented Jan 4, 2024

sgrabbit commented Feb 19, 2024

michal-rybinski commented Feb 21, 2024

ilyas1uphealth commented Mar 12, 2024 • edited

sidyag commented Mar 20, 2024

rajarshp commented Apr 1, 2024 • edited

michal-rybinski commented Apr 2, 2024 • edited

michal-rybinski commented Apr 4, 2024 • edited

michal-rybinski commented Apr 4, 2024

michal-rybinski commented Apr 4, 2024

sidyag commented Apr 4, 2024

michal-rybinski commented Apr 4, 2024

rajarshp commented Apr 8, 2024

sidyag commented Apr 8, 2024

michal-rybinski commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ilyas1uphealth commented Mar 12, 2024 •

edited

rajarshp commented Apr 1, 2024 •

edited

michal-rybinski commented Apr 2, 2024 •

edited

michal-rybinski commented Apr 4, 2024 •

edited