Unable to create replication for ECR on harbor running under kubernetes

[ramukima]

What can we help you?

I have harbor v2.2.2 running on a kubernetes cluster. I see the following error when I try to add an ECR endpoint without specifying access key/secret -

2021-05-28T06:08:54Z [ERROR] [/replication/adapter/awsecr/adapter.go:208]: failed to ping registry https://<id>.dkr.<region>.amazonaws.com: EC2RoleRequestError: EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I have added following environment variables to 'core' and 'jobservice' deployments -

• AWS_ROLE_ARN - Policies/Roles set in AWS to allow read access to my ECR instance
• AWS_WEB_IDENTITY_TOKEN_FILE - I am using kubernetes OIDC discovery and projected service account token
• AWS_REGION - Though it should not be required and is extracted from the registry URL.

I do not have any AWS metadata service running (e.g. KIAM) in the cluster. All IAM policies are in place in AWS for the role I am trying to use.

I tried an 'awscli' pod with similar environment variables and the aws CLI is able to talk to my ECR using the same role ARN.

I checked #13687 but no mention of what changes were done to the core/jobservice pods. Also, the setup explained in the issue uses KIAM to associate roles to pods, which is not my case.

Not sure why is the awecr adaptor trying to fetch roles from metadata despite having those environment variables set ?

[ramukima]

Reference: https://gist.github.com/stormcat24/e8172b4130776e486f2758508eb4f3aa

[bitsf]

if you don't enter access key, it will try to use IAM role, based on feature request #12553

[ramukima]

Following changes in src/pkg/reg/adapter/awsecr/auth.go (not the best) work for me (without having KIAM in my k8s cluster), note that I am trying this in non-EC2 environment.

func getAwsSvc(region, accessKey, accessSecret string, insecure bool, forceEndpoint *string) (*awsecrapi.ECR, error) {
	sess, err := session.NewSession()
	if err != nil {
		return nil, err
	}

	var tr *http.Transport
	if insecure {
		tr = commonhttp.GetHTTPTransport(commonhttp.InsecureTransport)
	} else {
		tr = commonhttp.GetHTTPTransport(commonhttp.SecureTransport)
	}

	var cred *credentials.Credentials
	log.Debugf("Aws Ecr getAuthorization %s", accessKey)
	if accessKey != "" {
		cred = credentials.NewStaticCredentials(
			accessKey,
			accessSecret,
			"")
	} else if os.Getenv("AWS_WEB_IDENTITY_TOKEN_FILE") != "" {
		aws.NewConfig().
			WithCredentialsChainVerboseErrors(true).
			WithCredentials(credentials.NewChainCredentials([]credentials.Provider{
				&credentials.EnvProvider{},
				&credentials.SharedCredentialsProvider{},
				// Required for IRSA
				stscreds.NewWebIdentityRoleProvider(
					sts.New(sess),
					os.Getenv("AWS_ROLE_ARN"),
					"harbor",
					os.Getenv("AWS_WEB_IDENTITY_TOKEN_FILE"),
				),
				&ec2rolecreds.EC2RoleProvider{
					Client: ec2metadata.New(sess),
				},
			})).
			WithHTTPClient(&http.Client{
				Transport: tr,
			}).
			WithRegion(region)
		cred = aws.NewConfig().Credentials

	} else {
		cred = ec2rolecreds.NewCredentials(sess)
	}

	config := &aws.Config{
		Credentials: cred,
		Region:      &region,
		HTTPClient: &http.Client{
			Transport: tr,
		},
	}
	if forceEndpoint != nil {
		config.Endpoint = forceEndpoint
	}

	svc := awsecrapi.New(sess, config)
	return svc, nil
}

[ramukima]

Is it worth getting a PR for this ?

[Geethree]

Is it worth getting a PR for this ?

I vote yes. Leveraging IAM Role Service accounts to let the aws sdk grab temporary credentials from a cluster's OIDC provider is handy.

[SeanKnight]

Related to #12888 which shouldn't have been closed.

[ramukima]

A single role attached to the pod via environment variable may not be perfect. Since the registries are added on demand in Harbor, deriving role_arn from the registry URL (account_id, region) and a predetermined ROLE_NAME could be an option. However, I am not sure how to achieve this - crossplane-contrib/provider-aws#597

[ramukima]

Example Patch - harbor_patch.txt

[DerrickMartinez]

Anyone going to get a PR in?

[DerrickMartinez]

@ramukima are you going to get a PR going?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.