Upgrade to 1.2 failed to load config file, /credentials/cloud #2191

SebastienTolron · 2020-01-14T08:17:48Z

What steps did you take and what happened:

I upgraded from 1.1 to 1.2 ( Followed https://velero.io/docs/v1.2.0/upgrade-to-1.2/ )

The velero pod crashed with error :

 error getting backup store for location "default": rpc error: code = Unknown desc = SharedConfigLoadError: failed to load config file, /credentials/cloud
caused by: INIParseError: invalid state with ASTKind { {4 NONE 0 [61]} true [{expr {1 STRING 0 [97 119 115 95 115 101 99 114 101 116 95 97 99 99 101 115 115 95 107 101 121]} true []}]} and TokenType sep

I double checked and seems that the secret still exists. Seems more like a parsing issue.

What did you expect to happen:

No regression

Environment:

Velero version (use velero version):

Client:
	Version: v1.2.0
	Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c

Velero features (use velero client config get features):

features: <NOT SET>

Kubernetes version (use kubectl version):

1.11.0

The text was updated successfully, but these errors were encountered:

skriss · 2020-01-16T16:51:49Z

@Stolr I've never seen that one before! What provider are you using? Is it possible to provide the credentials file with any sensitive information stripped out? I wonder if it's some kind of weird whitespace error.

SebastienTolron · 2020-01-17T07:37:46Z

@skriss : No issue with same config file using the v1.1.
Here is the credential file called credentials-velero

[default]
aws_access_key_id = XXX
aws_secret_access_key = XXX

With 1.1 , using :

velero install \
    --provider aws \
    --bucket velero \
    --use-restic \
    --secret-file ./credentials-velero  \
    --use-volume-snapshots=false \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://oca-miniolb.oca.local/

Working.

When upgrading : Not working ( see below )

When installing 1.2 from scratch

velero install \
    --provider aws \
    --bucket velero \
    --use-restic \
    --secret-file ./credentials-velero  \
    --use-volume-snapshots=false \
     --plugins velero/velero-plugin-for-aws:v1.0.0 \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://oca-miniolb.oca.local/

Same error


An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = SharedConfigLoadError: failed to load config file, /credentials/cloud
--
  | caused by: INIParseError: invalid state with ASTKind { {4 NONE 0 [61]} true [{expr {1 STRING 0 [97 119 115 95 115 101 99 114 101 116 95 97 99 99 101 115 115 95 107 101 121]} true []}]} and TokenType sep

I also tried to put a fake credentials to see and I got a proper error

An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid: rpc error: code = Unknown desc = InvalidAccessKeyId: The access key ID you provided does not exist in our records.
--
  | status code: 403, request id: 15EA9C544C2588A8, host id:

skriss · 2020-01-17T17:52:55Z

hmm - it looks pretty normal to me. Does anything change if you remove the whitespace around the two = signs? It looks valid as-is, so just experimenting..

SebastienTolron · 2020-01-20T07:14:41Z

@skriss : Same error :

credentials-velero

[default]
aws_access_key_id=XXX
aws_secret_access_key=XXX

Velero logs

time="2020-01-20T07:12:29Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:413"
--
  | An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = SharedConfigLoadError: failed to load config file, /credentials/cloud
  | caused by: INIParseError: invalid state with ASTKind { {4 NONE 0 [61]} true [{expr {1 STRING 0 [97 119 115 95 115 101 99 114 101 116 95 97 99 99 101 115 115 95 107 101 121]} true []}]} and TokenType sep

skriss · 2020-01-21T22:22:10Z

@Stolr I see a couple references to similar issues in the AWS SDK repo (e.g. aws/aws-sdk-go#2239) - anything like that seem relevant?

Does this credentials file work fine with the aws CLI?

It certainly seems to be related to the AWS SDK's parsing of the file. Perhaps you could generate a new access key and see if that fixes things?

carlisia · 2020-02-05T21:24:57Z

Hey @Stolr, has your issue been resolved?

SebastienTolron · 2020-02-10T09:01:46Z

Hey ,

Sorry for the late answer. I can't really re generat because it is already used.

I'll test with aws cli and let you know.

Still having the issue

SebastienTolron · 2020-02-11T13:54:32Z

Hey ,

Works fine with aws cli

 /usr/local/bin/aws --version
aws-cli/2.0.0 Python/3.7.3 Linux/4.19.0-6-amd64 botocore/2.0.0dev4

aws configure
AWS Access Key ID [None]: [OFUSCED]
AWS Secret Access Key [None]: [OFUSCED]
Default region name [None]:
Default output format [None]:

aws --endpoint-url http://oca-miniolb.oca.local/ s3 ls
2019-11-15 10:45:48 gitlab
2020-02-11 14:45:35 kibana-7
2019-10-31 10:42:06 velero

skriss · 2020-02-11T14:57:15Z

@Stolr OK, thanks for the info. We could try updating the version of the AWS SDK that we use to the latest, to see if that resolves it. If I publish a test image with that change for you, would you be able to try it out?

SebastienTolron · 2020-02-11T15:00:14Z

Sure , I'd love to

skriss · 2020-02-11T15:10:32Z

OK, I pushed steveheptio/velero-plugin-for-aws:latest-aws-sdk, which uses v1.28.14 of the AWS go SDK.

If you already have velero installed somewhere, you can do the following:

velero plugin remove velero/velero-plugin-for-aws:v1.0.0
velero plugin add steveheptio/velero-plugin-for-aws:latest-aws-sdk

Or you can kubectl -n velero edit deploy/velero, find the plugin container listed under initContainers, and change it manually.

Or, if you don't have velero up and running, you can use the new image as the value of the --plugins flag for velero install.

If you get an error again, please include the full log + error message! Thanks for helping us dig into this.

SebastienTolron · 2020-02-11T16:38:30Z

Ok so ,

Install velero from fresh instance

velero version
Client:
	Version: v1.2.0
	Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c

velero install \
    --provider aws \
    --bucket velero \
    --use-restic \
    --secret-file ./credentials-velero  \
    --use-volume-snapshots=false \
    --plugins velero/velero-plugin-for-aws:v1.0.0 \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://oca-miniolb.oca.local/ 


CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...

Namespace/velero: attempting to create resource
Namespace/velero: already exists, proceeding
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: already exists, proceeding
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
DaemonSet/restic: attempting to create resource
DaemonSet/restic: already exists, proceeding
DaemonSet/restic: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

==> ERROR ( same as above )

Then I edited the deployment to change the init container to your image

=> ERROR


  | time="2020-02-11T16:34:43Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:171"
-- | --
  | time="2020-02-11T16:34:43Z" level=info msg="Starting Velero server v1.2.0 (5d008491bbf681658d3e372da1a9d3a21ca4c03c)" logSource="pkg/cmd/server/server.go:173"
  | time="2020-02-11T16:34:43Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:177"
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
  | time="2020-02-11T16:34:43Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
  | time="2020-02-11T16:34:43Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:337" namespace=velero
  | time="2020-02-11T16:34:43Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:343" namespace=velero
  | time="2020-02-11T16:34:47Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:372"
  | time="2020-02-11T16:34:52Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:406"
  | time="2020-02-11T16:34:52Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:413"
  | An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = SharedConfigLoadError: failed to load config file, /credentials/cloud
  | caused by: INIParseError: invalid state with ASTKind { {4 NONE 0 [61]} true [{expr {1 STRING 0 [97 119 115 95 115 101 99 114 101 116 95 97 99 99 101 115 115 95 107 101 121]} true []}]} and TokenType sep

What king of log do you need ?

I'll try to create a whole new instance of minio and changing access key tomorrow and test again

I've got some caracters ] for example that might also cause the issue

skriss · 2020-02-11T16:50:00Z

Argh. You should probably file an issue in the https://github.com/aws/aws-sdk-go repo, as it seems pretty likely that there's a bug in there. I would include the fact that the file works OK with the aws CLI, but not the go SDK. Not exactly sure if/how they'll be able to debug without having the details of the credentials file, but maybe some incomplete info about special chars etc. will help.

SebastienTolron · 2020-02-12T07:18:26Z

Hey @skriss thanks for your feedback.

Are you able to reproduce on your side with my credentials ?

[default]
aws_access_key_id=sPrBwkFKD8Lsp
aws_secret_access_key=]hVm3bs2JaZygh672

I just secured my instance so not a problem to give you theses keys

If you can give me a feedback if you are able to reproduce that would be awesome.

Thanks

skriss · 2020-02-12T15:49:32Z

@Stolr yep, I was able to reproduce. However, I did find a workaround - putting double quotes around the secret access key value allows it to parse, i.e.:

[default]
aws_access_key_id=sPrBwkFKD8Lsp
aws_secret_access_key="]hVm3bs2JaZygh672"

Sure looks like that leading ] in the access key is messing up the parser in the AWS go SDK.

SebastienTolron · 2020-02-12T17:36:06Z

Awesome !

I should have think of this workaround !

Maybe the documentation can be updated to put that doubles quotes to make sure this will work for Everyone.

@skriss I'll close the issue. Thanks a lot for your help.

snorwin · 2021-06-28T08:26:20Z

I was also facing the same issue and opened an issue on the AWS SDK project: aws/aws-sdk-go#3958

snorwin · 2021-07-02T08:50:36Z

@Stolr the issue aws/aws-sdk-go#3958 is now fixed maybe can you upgrade to the its latest version.

SebastienTolron changed the title ~~failed to load config file, /credentials/cloud~~ Upgrade to 1.2 failed to load config file, /credentials/cloud Jan 14, 2020

skriss added Needs investigation Needs info Waiting for information labels Jan 16, 2020

SebastienTolron closed this as completed Feb 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to 1.2 failed to load config file, /credentials/cloud #2191

Upgrade to 1.2 failed to load config file, /credentials/cloud #2191

SebastienTolron commented Jan 14, 2020

skriss commented Jan 16, 2020

SebastienTolron commented Jan 17, 2020

skriss commented Jan 17, 2020 •

edited

SebastienTolron commented Jan 20, 2020

skriss commented Jan 21, 2020 •

edited

carlisia commented Feb 5, 2020

SebastienTolron commented Feb 10, 2020

SebastienTolron commented Feb 11, 2020

skriss commented Feb 11, 2020

SebastienTolron commented Feb 11, 2020

skriss commented Feb 11, 2020 •

edited

SebastienTolron commented Feb 11, 2020

skriss commented Feb 11, 2020

SebastienTolron commented Feb 12, 2020

skriss commented Feb 12, 2020

SebastienTolron commented Feb 12, 2020

snorwin commented Jun 28, 2021

snorwin commented Jul 2, 2021

Upgrade to 1.2 failed to load config file, /credentials/cloud #2191

Upgrade to 1.2 failed to load config file, /credentials/cloud #2191

Comments

SebastienTolron commented Jan 14, 2020

skriss commented Jan 16, 2020

SebastienTolron commented Jan 17, 2020

skriss commented Jan 17, 2020 • edited

SebastienTolron commented Jan 20, 2020

skriss commented Jan 21, 2020 • edited

carlisia commented Feb 5, 2020

SebastienTolron commented Feb 10, 2020

SebastienTolron commented Feb 11, 2020

skriss commented Feb 11, 2020

SebastienTolron commented Feb 11, 2020

skriss commented Feb 11, 2020 • edited

SebastienTolron commented Feb 11, 2020

skriss commented Feb 11, 2020

SebastienTolron commented Feb 12, 2020

skriss commented Feb 12, 2020

SebastienTolron commented Feb 12, 2020

snorwin commented Jun 28, 2021

snorwin commented Jul 2, 2021

skriss commented Jan 17, 2020 •

edited

skriss commented Jan 21, 2020 •

edited

skriss commented Feb 11, 2020 •

edited