Flink Application Start Failed. Reason Unable to execute HTTP request: Connect to secretsmanager.us-east-1.amazonaws.com:443 #3023
Comments
cd-ujangiti
added
bug
|
You need to make sure the VPC has access to the SecretsManager endpoint. You can achieve this by configuring an internet gateway to connect your VPC to the internet, or you can use a VPC endpoint to connect to AWS services privately. Check the VPC Developer Guide for more detailed info on the options: https://docs.aws.amazon.com/vpc/latest/userguide/extend-intro.html |
debora-ito
added
closing-soon
|
Hi @debora-ito , This is resolved after creating the AWS Secrets Manager VPC endpoint with public subnets. Using an AWS Secrets Manager VPC endpoint |
COMMENT VISIBILITY WARNINGComments on closed issues are hard for our team to see. |
Describe the bug
I have same issue while connecting from kinesis data analytics to RDS MYSQL through secrets manager. I have provided secrets manager full access, still connectivity failing with same below error. flink application able to connect DB with username and password but not through secrets manager.
Details: RDS MySQL and Kinesis data analytics are running on same VPC and private subnets.
Kinesis data analytics flink app should able to connect MySQL database through secrets manager. I'm storing the db credentials in secrets manager and add that secret name in flink runtime properties.
----------Error logs-------
Flink Application Start Failed. Reason Unable to execute HTTP request: Connect to secretsmanager.us-east-1.amazonaws.com:443 [secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY] failed: connect timed out
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.doInvoke(AWSSecretsManagerClient.java:2737)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.invoke(AWSSecretsManagerClient.java:2704)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.invoke(AWSSecretsManagerClient.java:2693)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.executeDescribeSecret(AWSSecretsManagerClient.java:908)
at com.amazonaws.services.secretsmanager.AWSSecretsManagerClient.describeSecret(AWSSecretsManagerClient.java:878)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheItem.executeRefresh(SecretCacheItem.java:102)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheItem.executeRefresh(SecretCacheItem.java:32)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheObject.refresh(SecretCacheObject.java:188)
at com.amazonaws.secretsmanager.caching.cache.SecretCacheObject.getSecretValue(SecretCacheObject.java:286)
at com.amazonaws.secretsmanager.caching.SecretCache.getSecretString(SecretCache.java:123)
at com.cd.ftr.util.SecretValueProvider.getMysqlDetailsFromSecretManager(SecretValueProvider.java:36)
at com.cd.ftr.DeviceMessageStreaming.buildMysqlProperties(DeviceMessageStreaming.java:179)
at com.cd.ftr.DeviceMessageStreaming.run(DeviceMessageStreaming.java:55)
at com.cd.ftr.DeviceMessageStreaming.main(DeviceMessageStreaming.java:34)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
at org.apache.flink.runtime.webmonitor.handlers.JarRunOverrideHandler.lambda$handleRequest$3(JarRunOverrideHandler.java:238)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to secretsmanager.us-east-1.amazonaws.com:443 [secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY, secretsmanager.us-east-1.amazonaws.com/IP.ADDRESS.XX.YY] failed: connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy54.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
... 38 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:609)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 54 more
Expected Behavior
Kinesis data analytics flink app should able to connect MyQsl database through secrets manager. I'm storing the db credentials in secrets manager and add that secrets name in flink runtime properties.
Current Behavior
kinesis data analytics flink is not able to connect DB through secrets manager.
Reproduction Steps
NA
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
JDK version used
JDK 11
Operating System and version
NA
The text was updated successfully, but these errors were encountered: