aws components - validate credentials and connection on init or ping

[famarting]

Describe the feature

Existing component implementations such as postgres, mysql, redis,..., use init function or at least the optional ping function in order to validate as much as possible the provided configuration for the component and to guarantee that it's properly configured and it would work as expected during runtime.

To get a consistent experience, and to be able to properly validate components and avoid runtime errors because of misconfigurations, It would be desirable that as much components as possible properly validate the provided configuration and establish a connection, or at least somehow probe the connection, with the underlying backing infrastructure.

The ask for this issue is to try to implement that behavior for the aws components.

i.e I've had issues with the aws s3 binding component. I had an invalid configuration, with wrong access key and secret key, however the component initialization was successfull but during runtime I got this error using the binding API

{
    "errorCode": "ERR_INVOKE_OUTPUT_BINDING",
    "message": "error invoking output binding bucket-wrong-users: s3 binding error: uploading failed: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id:xxxxx, host id: xxxxxxx"
  },

I assume there should be some aws API or function in the aws sdk which can be used to probe aws and test the provided configuration, credentials,.... If no adhoc validating API exists, maybe the component can perform a GET request to the s3 bucket or other metadata API... however I'm afraid that may become expensive for users.

Release Note

RELEASE NOTE: ADD connection validation for aws components

RELEASE NOTE:

[elena-kolevska]

/assign

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[elena-kolevska]

@famarting we can easily check if the credentials are valid using sts.GetCallerIdentity. This will be shared across all aws components.
In addition, we could do an extra check specific to the component, for example "Is the DynamoDb table present"? I couldn't find any specific api for that check, but we could do a SCAN with a limit of 1. Should I add that?

Also, can you please reopen the issue?

[famarting]

hmm, I wonder if sts.GetCallerIdentity will help or cause trouble, because I believe your aws user will have to have the permissions to perform that sts call. Maybe component specialized checks, such as the scan with limit of 1 that you mentioned, are more helpful and likely to work better in most situations?

The only downside is that it will need a different implementation for each different component... s3 will need something different than sqs or dynamodb...

[elena-kolevska]

So that's actually part of why I suggested the sts.GetCallerIdentity call, because it doesn't require permissions, as per the docs:

No permissions are required to perform this operation. If an administrator attaches a policy to your identity that explicitly denies access to the sts:GetCallerIdentity action, you can still perform this operation. Permissions are not required because the same information is returned when access is denied.

[famarting]

oh nice, good catch, ty, so then the perfect solution would be calling first to sts.GetCallerIdentity and then the specialized validation

[elena-kolevska]

@famarting I ended up removing the sts.GetCallerIdentity check because it's redundant if we're doing the specialised checks anyway.
I will do this for other components too.
Here's the PR: #3285

[elena-kolevska]

PR for AWS secret stores: #3301

[elena-kolevska]

Still missing:

• Bindings: DynamoDB
• Bindings: Kinesis
• Bindings: S3
• Bindings: SES
• Bindings: SNS
• Bindings: SQS

I'll be working on them this week