remote backend does not allow imports with sensitive remote variables

[petems]

Current Terraform Version

0.13.4

Terraform Configuration Files

terraform {
  backend "remote" {
    organization = "psouter-hashicorp"

    workspaces {
      name = "terraform_tfvars_import"
    }
  }
}

variable "access_key" {}
variable "secret_key" {}
variable "region" {}

provider "aws" {
  access_key = var.access_key
  secret_key = var.secret_key
  region = var.region
}

resource "aws_eip" "bar" {}

Debug Output

Note: Same logs for all attempts:

• No variables locally - Fails
• local file sensitive.auto.tfvars with the appropriate values - Ignored
• -var and -var-file arguments - Ignored
• TF_VAR_ environment variables - Ignored

CLICK ME

TF_LOG=TRACE terraform import aws_eip.e eipalloc-123example
2020/10/06 21:37:58 [INFO] Terraform version: 0.13.4
2020/10/06 21:37:58 [INFO] Go runtime version: go1.14.7
2020/10/06 21:37:58 [INFO] CLI args: []string{"/usr/local/Cellar/tfenv/1.0.2/versions/0.13.4/terraform", "import", "aws_eip.e", "eipalloc-123example"}
2020/10/06 21:37:58 [DEBUG] Attempting to open CLI config file: /Users/petersouter/.terraformrc
2020/10/06 21:37:58 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/06 21:37:58 Loading CLI configuration from /Users/petersouter/.terraform.d/credentials.tfrc.json
2020/10/06 21:37:58 [DEBUG] checking for credentials in "/Users/petersouter/.terraform.d/plugins"
2020/10/06 21:37:58 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/06 21:37:58 [DEBUG] will search for provider plugins in /Users/petersouter/.terraform.d/plugins
2020/10/06 21:37:58 [TRACE] getproviders.SearchLocalDirectory: /Users/petersouter/.terraform.d/plugins is a symlink to /Users/petersouter/.terraform.d/plugins
2020/10/06 21:37:58 [WARN] Provider plugin search ignored symlink /Users/petersouter/.terraform.d/plugins/terraform-provider-awsdirectoryradius: only the base directory /Users/petersouter/.terraform.d/plugins may be a symlink
2020/10/06 21:37:58 [WARN] Provider plugin search ignored symlink /Users/petersouter/.terraform.d/plugins/terraform-provider-sshconfig: only the base directory /Users/petersouter/.terraform.d/plugins may be a symlink
2020/10/06 21:37:58 [DEBUG] ignoring non-existing provider search directory /Users/petersouter/Library/Application Support/io.terraform/plugins
2020/10/06 21:37:58 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/10/06 21:37:58 [INFO] CLI command args: []string{"import", "aws_eip.e", "eipalloc-123example"}
2020/10/06 21:37:58 [TRACE] Meta.Backend: built configuration for "remote" backend with hash value 631411100
2020/10/06 21:37:58 [TRACE] Preserving existing state lineage "fd6ca71a-dbc4-a0c8-f701-cac5301005be"
2020/10/06 21:37:58 [TRACE] Preserving existing state lineage "fd6ca71a-dbc4-a0c8-f701-cac5301005be"
2020/10/06 21:37:58 [TRACE] Meta.Backend: working directory was previously initialized for "remote" backend
2020/10/06 21:37:58 [TRACE] Meta.Backend: using already-initialized, unchanged "remote" backend configuration
2020/10/06 21:37:58 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2020/10/06 21:37:58 [TRACE] HTTP client GET request to https://app.terraform.io/.well-known/terraform.json
2020/10/06 21:37:59 [DEBUG] Retrieve version constraints for service tfe.v2.1 and product terraform
2020/10/06 21:37:59 [TRACE] HTTP client GET request to https://checkpoint-api.hashicorp.com/v1/versions/tfe.v2.1?product=terraform
2020/10/06 21:37:59 [TRACE] Meta.Backend: instantiated backend of type *remote.Remote
2020/10/06 21:37:59 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/10/06 21:37:59 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
2020/10/06 21:37:59 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/aws v3.9.0 for darwin_amd64 at .terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64
2020/10/06 21:37:59 [TRACE] providercache.fillMetaCache: including .terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/aws 3.9.0
2020/10/06 21:38:00 [DEBUG] checking for provisioner in "."
2020/10/06 21:38:00 [DEBUG] checking for provisioner in "/usr/local/Cellar/tfenv/1.0.2/versions/0.13.4"
2020/10/06 21:38:00 [DEBUG] checking for provisioner in "/Users/petersouter/.terraform.d/plugins"
2020/10/06 21:38:00 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/10/06 21:38:00 [TRACE] Meta.Backend: backend *remote.Remote supports operations
2020/10/06 21:38:00 [TRACE] backend/remote: requesting state manager for workspace "terraform_tfvars_import"
2020/10/06 21:38:00 [TRACE] backend/remote: requesting state lock for workspace "terraform_tfvars_import"
2020/10/06 21:38:00 [TRACE] backend/remote: reading remote state for workspace "terraform_tfvars_import"
2020/10/06 21:38:00 [TRACE] backend/remote: retrieving remote state snapshot for workspace "terraform_tfvars_import"
2020/10/06 21:38:00 [TRACE] backend/remote: loading configuration for the current working directory
2020/10/06 21:38:00 [TRACE] backend/remote: looking up workspace id for psouter-hashicorp/terraform_tfvars_import
2020/10/06 21:38:01 [TRACE] backend/remote: retrieving variables from workspace terraform_tfvars_import/psouter-hashicorp (ws-NC7oKJxyqBorWLdR)
2020/10/06 21:38:01 [TRACE] terraform.NewContext: starting
2020/10/06 21:38:01 [TRACE] terraform.NewContext: loading provider schemas
2020/10/06 21:38:01 [TRACE] LoadSchemas: retrieving schema for provider type "registry.terraform.io/hashicorp/aws"
2020-10-06T21:38:01.443+0100 [INFO]  plugin: configuring client automatic mTLS
2020-10-06T21:38:01.470+0100 [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5 args=[.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5]
2020-10-06T21:38:01.479+0100 [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5 pid=15468
2020-10-06T21:38:01.480+0100 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5
2020-10-06T21:38:01.522+0100 [INFO]  plugin.terraform-provider-aws_v3.9.0_x5: configuring server automatic mTLS: timestamp=2020-10-06T21:38:01.522+0100
2020-10-06T21:38:01.551+0100 [DEBUG] plugin.terraform-provider-aws_v3.9.0_x5: plugin address: network=unix address=/var/folders/n4/26ry5rkn03l_51jyr74_38lc0000gp/T/plugin683576411 timestamp=2020-10-06T21:38:01.551+0100
2020-10-06T21:38:01.551+0100 [DEBUG] plugin: using plugin: version=5
2020/10/06 21:38:01 [TRACE] GRPCProvider: GetSchema
2020-10-06T21:38:01.603+0100 [TRACE] plugin.stdio: waiting for stdio data
2020/10/06 21:38:01 [TRACE] GRPCProvider: Close
2020-10-06T21:38:01.740+0100 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-10-06T21:38:01.742+0100 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5 pid=15468
2020-10-06T21:38:01.742+0100 [DEBUG] plugin: plugin exited
2020/10/06 21:38:01 [TRACE] terraform.NewContext: complete
2020/10/06 21:38:01 [TRACE] backend/remote: finished building terraform.Context
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.ConfigTransformer
2020/10/06 21:38:01 [TRACE] ConfigTransformer: Starting for path:
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.ConfigTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.AttachResourceConfigTransformer
2020/10/06 21:38:01 [TRACE] AttachResourceConfigTransformer: attaching to "aws_eip.e" (*terraform.NodeAbstractResource) config from /Users/petersouter/projects/terraform_tfvars_import/main.tf:21,1-23
2020/10/06 21:38:01 [TRACE] AttachResourceConfigTransformer: attaching provider meta configs to aws_eip.e
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.AttachResourceConfigTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.ImportStateTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.ImportStateTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.RootVariableTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.RootVariableTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.graphTransformerMulti
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Executing graph transform *terraform.ProviderConfigTransformer
2020/10/06 21:38:01 [TRACE] ProviderConfigTransformer: attaching to "provider[\"registry.terraform.io/hashicorp/aws\"]" provider configuration from /Users/petersouter/projects/terraform_tfvars_import/main.tf:15,1-15
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Completed graph transform *terraform.ProviderConfigTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Executing graph transform *terraform.MissingProviderTransformer
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Completed graph transform *terraform.MissingProviderTransformer (no changes)
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Executing graph transform *terraform.ProviderTransformer
2020/10/06 21:38:01 [TRACE] ProviderTransformer: exact match for provider["registry.terraform.io/hashicorp/aws"] serving aws_eip.e
2020/10/06 21:38:01 [DEBUG] ProviderTransformer: "aws_eip.e" (*terraform.NodeAbstractResource) needs provider["registry.terraform.io/hashicorp/aws"]
2020/10/06 21:38:01 [TRACE] ProviderTransformer: exact match for provider["registry.terraform.io/hashicorp/aws"] serving aws_eip.e (import id "eipalloc-123example")
2020/10/06 21:38:01 [DEBUG] ProviderTransformer: "aws_eip.e (import id \"eipalloc-123example\")" (*terraform.graphNodeImportState) needs provider["registry.terraform.io/hashicorp/aws"]
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Completed graph transform *terraform.ProviderTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Executing graph transform *terraform.PruneProviderTransformer
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Completed graph transform *terraform.PruneProviderTransformer (no changes)
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Executing graph transform *terraform.ParentProviderTransformer
2020/10/06 21:38:01 [TRACE] (graphTransformerMulti) Completed graph transform *terraform.ParentProviderTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.graphTransformerMulti with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.LocalTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.LocalTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.OutputTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.OutputTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.ModuleVariableTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.ModuleVariableTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.AttachSchemaTransformer
2020/10/06 21:38:01 [TRACE] AttachSchemaTransformer: attaching resource schema to aws_eip.e
2020/10/06 21:38:01 [TRACE] AttachSchemaTransformer: attaching provider config schema to provider["registry.terraform.io/hashicorp/aws"]
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.AttachSchemaTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.ModuleExpansionTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.ModuleExpansionTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.ReferenceTransformer
2020/10/06 21:38:01 [DEBUG] ReferenceTransformer: "aws_eip.e" references: []
2020/10/06 21:38:01 [DEBUG] ReferenceTransformer: "aws_eip.e (import id \"eipalloc-123example\")" references: []
2020/10/06 21:38:01 [DEBUG] ReferenceTransformer: "var.access_key" references: []
2020/10/06 21:38:01 [DEBUG] ReferenceTransformer: "var.secret_key" references: []
2020/10/06 21:38:01 [DEBUG] ReferenceTransformer: "var.region" references: []
2020/10/06 21:38:01 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/aws\"]" references: [var.secret_key var.access_key var.region]
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.ReferenceTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
    var.access_key - *terraform.NodeRootVariable
    var.region - *terraform.NodeRootVariable
    var.secret_key - *terraform.NodeRootVariable
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.attachDataResourceDependenciesTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.attachDataResourceDependenciesTransformer (no changes)
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.CloseProviderTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.CloseProviderTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
    var.access_key - *terraform.NodeRootVariable
    var.region - *terraform.NodeRootVariable
    var.secret_key - *terraform.NodeRootVariable
  provider["registry.terraform.io/hashicorp/aws"] (close) - *terraform.graphNodeCloseProvider
    aws_eip.e - *terraform.NodeAbstractResource
    aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.CloseRootModuleTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.CloseRootModuleTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
    var.access_key - *terraform.NodeRootVariable
    var.region - *terraform.NodeRootVariable
    var.secret_key - *terraform.NodeRootVariable
  provider["registry.terraform.io/hashicorp/aws"] (close) - *terraform.graphNodeCloseProvider
    aws_eip.e - *terraform.NodeAbstractResource
    aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  root - *terraform.nodeCloseModule
    provider["registry.terraform.io/hashicorp/aws"] (close) - *terraform.graphNodeCloseProvider
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [TRACE] Executing graph transform *terraform.TransitiveReductionTransformer
2020/10/06 21:38:01 [TRACE] Completed graph transform *terraform.TransitiveReductionTransformer with new graph:
  aws_eip.e - *terraform.NodeAbstractResource
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
    provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
  provider["registry.terraform.io/hashicorp/aws"] - *terraform.NodeApplyableProvider
    var.access_key - *terraform.NodeRootVariable
    var.region - *terraform.NodeRootVariable
    var.secret_key - *terraform.NodeRootVariable
  provider["registry.terraform.io/hashicorp/aws"] (close) - *terraform.graphNodeCloseProvider
    aws_eip.e - *terraform.NodeAbstractResource
    aws_eip.e (import id "eipalloc-123example") - *terraform.graphNodeImportState
  root - *terraform.nodeCloseModule
    provider["registry.terraform.io/hashicorp/aws"] (close) - *terraform.graphNodeCloseProvider
  var.access_key - *terraform.NodeRootVariable
  var.region - *terraform.NodeRootVariable
  var.secret_key - *terraform.NodeRootVariable
  ------
2020/10/06 21:38:01 [DEBUG] Starting graph walk: walkImport
2020/10/06 21:38:01 [TRACE] dag/walk: visiting "var.secret_key"
2020/10/06 21:38:01 [TRACE] vertex "var.secret_key": starting visit (*terraform.NodeRootVariable)
2020/10/06 21:38:01 [TRACE] vertex "var.secret_key": evaluating
2020/10/06 21:38:01 [TRACE] dag/walk: visiting "var.access_key"
2020/10/06 21:38:01 [TRACE] [walkImport] Entering eval tree: var.secret_key
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalSequence
2020/10/06 21:38:01 [TRACE] [walkImport] Exiting eval tree: var.secret_key
2020/10/06 21:38:01 [TRACE] vertex "var.secret_key": visit complete
2020/10/06 21:38:01 [TRACE] vertex "var.access_key": starting visit (*terraform.NodeRootVariable)
2020/10/06 21:38:01 [TRACE] dag/walk: visiting "var.region"
2020/10/06 21:38:01 [TRACE] vertex "var.access_key": evaluating
2020/10/06 21:38:01 [TRACE] vertex "var.region": starting visit (*terraform.NodeRootVariable)
2020/10/06 21:38:01 [TRACE] [walkImport] Entering eval tree: var.access_key
2020/10/06 21:38:01 [TRACE] vertex "var.region": evaluating
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalSequence
2020/10/06 21:38:01 [TRACE] [walkImport] Entering eval tree: var.region
2020/10/06 21:38:01 [TRACE] [walkImport] Exiting eval tree: var.access_key
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalSequence
2020/10/06 21:38:01 [TRACE] vertex "var.access_key": visit complete
2020/10/06 21:38:01 [TRACE] [walkImport] Exiting eval tree: var.region
2020/10/06 21:38:01 [TRACE] vertex "var.region": visit complete
2020/10/06 21:38:01 [TRACE] dag/walk: visiting "provider[\"registry.terraform.io/hashicorp/aws\"]"
2020/10/06 21:38:01 [TRACE] vertex "provider[\"registry.terraform.io/hashicorp/aws\"]": starting visit (*terraform.NodeApplyableProvider)
2020/10/06 21:38:01 [TRACE] vertex "provider[\"registry.terraform.io/hashicorp/aws\"]": evaluating
2020/10/06 21:38:01 [TRACE] [walkImport] Entering eval tree: provider["registry.terraform.io/hashicorp/aws"]
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalSequence
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalInitProvider
2020-10-06T21:38:01.743+0100 [INFO]  plugin: configuring client automatic mTLS
2020-10-06T21:38:01.768+0100 [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5 args=[.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5]
2020-10-06T21:38:01.778+0100 [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5 pid=15469
2020-10-06T21:38:01.778+0100 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5
2020-10-06T21:38:01.814+0100 [INFO]  plugin.terraform-provider-aws_v3.9.0_x5: configuring server automatic mTLS: timestamp=2020-10-06T21:38:01.813+0100
2020-10-06T21:38:01.842+0100 [DEBUG] plugin.terraform-provider-aws_v3.9.0_x5: plugin address: address=/var/folders/n4/26ry5rkn03l_51jyr74_38lc0000gp/T/plugin215886296 network=unix timestamp=2020-10-06T21:38:01.842+0100
2020-10-06T21:38:01.842+0100 [DEBUG] plugin: using plugin: version=5
2020/10/06 21:38:01 [TRACE] BuiltinEvalContext: Initialized "provider[\"registry.terraform.io/hashicorp/aws\"]" provider for provider["registry.terraform.io/hashicorp/aws"]
2020/10/06 21:38:01 [TRACE] eval: terraform.EvalNoop
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalOpFilter
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalSequence
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalGetProvider
2020/10/06 21:38:01 [TRACE] eval: terraform.EvalNoop
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalOpFilter
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalSequence
2020/10/06 21:38:01 [TRACE] eval: *terraform.EvalConfigProvider
2020-10-06T21:38:01.894+0100 [TRACE] plugin.stdio: waiting for stdio data
2020/10/06 21:38:01 [TRACE] buildProviderConfig for provider["registry.terraform.io/hashicorp/aws"]: using explicit config only
2020/10/06 21:38:01 [TRACE] GRPCProvider: GetSchema
2020/10/06 21:38:02 [WARN] eval: *terraform.EvalConfigProvider, non-fatal err: Invalid provider configuration: The configuration for provider["registry.terraform.io/hashicorp/aws"] depends on values that cannot be determined until apply.
2020/10/06 21:38:02 [ERROR] eval: *terraform.EvalSequence, err: Invalid provider configuration: The configuration for provider["registry.terraform.io/hashicorp/aws"] depends on values that cannot be determined until apply.
2020/10/06 21:38:02 [ERROR] eval: *terraform.EvalOpFilter, err: Invalid provider configuration: The configuration for provider["registry.terraform.io/hashicorp/aws"] depends on values that cannot be determined until apply.
2020/10/06 21:38:02 [ERROR] eval: *terraform.EvalSequence, err: Invalid provider configuration: The configuration for provider["registry.terraform.io/hashicorp/aws"] depends on values that cannot be determined until apply.
2020/10/06 21:38:02 [TRACE] [walkImport] Exiting eval tree: provider["registry.terraform.io/hashicorp/aws"]
2020/10/06 21:38:02 [TRACE] vertex "provider[\"registry.terraform.io/hashicorp/aws\"]": visit complete
2020/10/06 21:38:02 [TRACE] dag/walk: upstream of "aws_eip.e" errored, so skipping
2020/10/06 21:38:02 [TRACE] dag/walk: upstream of "aws_eip.e (import id \"eipalloc-123example\")" errored, so skipping
2020/10/06 21:38:02 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2020/10/06 21:38:02 [TRACE] dag/walk: upstream of "root" errored, so skipping

Warning: Value for var.secret_key unavailable

The value of variable "secret_key" is marked as sensitive in the remote
workspace. This operation always runs locally, so the value for that variable
is not available.


Warning: Value for var.access_key unavailable

The value of variable "access_key" is marked as sensitive in the remote
workspace. This operation always runs locally, so the value for that variable
is not available.


Error: Invalid provider configuration

  on /Users/petersouter/projects/terraform_tfvars_import/main.tf line 15:
  15: provider "aws" {

The configuration for provider["registry.terraform.io/hashicorp/aws"] depends
on values that cannot be determined until apply.

2020-10-06T21:38:02.205+0100 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-10-06T21:38:02.208+0100 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.9.0/darwin_amd64/terraform-provider-aws_v3.9.0_x5 pid=15469
2020-10-06T21:38:02.208+0100 [DEBUG] plugin: plugin exited

Expected Behavior

terraform import aws_eip.e eipalloc-123example
aws_eip.e: Importing from ID "eipalloc-123example"...
aws_eip.e: Import prepared!
  Prepared aws_eip for import
aws_eip.e: Refreshing state... [id=eipalloc-123example]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Actual Behavior

$ terraform import aws_eip.bar eipalloc-eipalloc-00a10e96

Warning: Value for var.secret_key unavailable

The value of variable "secret_key" is marked as sensitive in the remote
workspace. This operation always runs locally, so the value for that variable
is not available.


Warning: Value for var.access_key unavailable

The value of variable "access_key" is marked as sensitive in the remote
workspace. This operation always runs locally, so the value for that variable
is not available.


Error: Invalid provider configuration

  on /Users/petersouter/projects/terraform_tfvars_import/main.tf line 15:
  15: provider "aws" {

The configuration for provider["registry.terraform.io/hashicorp/aws"] depends
on values that cannot be determined until apply.

Steps to Reproduce

• Create remote variables marked as Sensitive
• Set a backend remote using the workspace with the Sensitive variables
• Try to run an import locally

Additional Context

Support for reading remote backend variables for local operations was added in Terraform 0.12.13.

Variables marked as sensitive in TFE cannot have their values retrieved via the API. So for a local operation such as import, the variables are set with empty or null placeholder values so that some kinds of local operations can succeed.

However, there's currently not a way to overwrite these null/empty placeholder values with the standard CLI options (eg. -var, -var-file, TF_VAR_ env setting)

Relevant code:

terraform/backend/remote/backend_context.go

Line 212 in bad0adb

case v.definition.Sensitive:

8f27409

References

Related issue from the TFE Provider:

• tfe_workspace import breaks when using terraform token in variable and remote execution terraform-provider-tfe#93

[Schoonology]

@pkolyvas If this is considered an enhancement, is there a blessed workaround?

We've migrated our team to Terraform Cloud for state management, but are still importing some legacy infrastructure, and this prevents us from doing so without some really dangerous, manual state modification.

[nikolay]

@Schoonology It surely is a bug. The only workaround possible without a code change is:

1. Lock the workspace
2. Delete the sensitive variable(s) remotely
3. Create a file like temp.auto.tfvats, set value(s) using (an) environmental variable(s), or pass them in config: anonymous cluster blocks in a service #4 via the CLI arguments to set those sensitive values locally
4. Perform the necessary terraform imports
5. Undo local changes and recreate the sensitive variables remotely
6. Unlock the workspace

I've been pulling hairs for months after we switched to TFC and this above is the only thing that works. I tried many others things like setting default values locally, terraform.tfvars, and .auto.tfvars files - without deleting the sensitive value remotely, nothing else works!

[nikolay]

@pkolyvas As our TFC plans grew big, we can't use terraform import anymore, which is really sad. Is this something you're gonna fix anytime soon or should we work on an internal tool to go around this?

[nikolay]

From what I see, this is not solved in v0.14 (as it's not listed in the changelog), we hope it at least gets fixed in v0.15. Most of the people experiencing these issues are paid customers using TFE, TFC, or the upcoming HashiCorp Cloud. It's really sad that we have to delete all sensitive variables before importing resources and then re-add them as this is the only workaround (or maybe pulling the state locally, removing the backend, grabbing those sensitive values if we have access to them, adding them to a tfvars file, and then undoing all the code surgery and pushing the state to TFC). Is there a particular reason why imports can't be done remotely via the backend?

[mltsy]

I agree that this is a bug, not an enhancement. (But I guess maybe our definitions are different than Hashicorp's?)

How are we supposed to import any existing infrastructure into TF cloud like this? Seems like it should just automatically override sensitive remote values with anything provided locally during import (unless there's a surprise TF Cloud import feature coming! Although I feel like this might also apply to other commands?)

[mltsy]

One alternative solution to at least provide a sensible workaround would be to allow us to mark a variable in TF cloud as "disabled" so that we don't have to delete and re-create the sensitive data (which is the most cumbersome part)

[datfinesoul]

At a minimum, we should be able to use the -var or -var-file CLI arguments to override the use of the remotely stored sensitive variables. Not being able to import at all due to the existence of remote variables seems like a bug.

Using TF Cloud, the only way I've gotten around this is to:

• Pull the latest terraform source locally
• Change the Execution Mode to Local in Settings->General on TF Cloud
• Locally turn off the remote backend (eg. mv backend.tf backend.tf.off)
• tf init
• Run imports
• Locally turn on the remote backend (eg. mv backend.tf backend.tf.off)
• tf init
• Check in the changes terraform code into source control
• Change the Execution Mode to Remote in Settings->General on TF Cloud

This "worked", but it basically defeats the purpose of sensitive remote variables and the use of the cloud for locks and version control.

Edit: I'm using terraform 0.14.4

[jbg]

@pkolyvas Respectfully, I think marking this as an enhancement was surely a mistake. This seems quite clearly a bug: it's not possible to use terraform import any more in the (very common) case that you have remote state and at least one sensitive variable. I am worried that the lack of traction this issue has got in the last several months is due to it being marked as an enhancement.

The workarounds documented in this issue are not workable in a large team with many changes happening, as they bypass locks and version control.

[alexandresalome]

I got around this issue by doing the following:

• Run terraform state pull > terraform.tfstate
• Delete .terraform directory
• Comment the backend "remote"
• Run terraform init
• Run terraform state list to make sure it's correct locally
• Run terraform import XXX
• Run terraform state list, because I'm scared of breaking things
• Uncomment the backend "remote"
• Delete .terraform directory (I love it)
• Run terraform init
• Compare with diff the 2 files indicated in the prompt to make sure the change is EXACTLY what I want.
• Answer "yes" ❤️

[datfinesoul]

I completely agree with @jbg, all of the workarounds we have tried on our team seem to be clunky at best. The import command needs to behave more like plan and apply in terms of how handles remote state and sensitive variables. Imports are key to working with larger infrastructure setups, and trying to migrate existing infrastructure to terraform. Marking this as a bug as opposed to an enhancement seems more appropriate.

[ajbouh]

Another workaround from #26549 is to manually (and temporarily) replace all references to sensitive variables with the secret values directly.

This is obviously dangerous but may have fewer moving parts than the suggestions above. Hard to see how this is not a bug due to unintended interactions of:

• how import works
• how remote state security works for sensitive variables
• patterns that providers use for access to secrets

[endorama]

I faced the same issue, and the situation is grim. Due to all the checks terraform performs on variables (presence or absence) every workaround based on variables to this issue is a major pain and require extensive understanding of Terraform internals.

If you're a paying customer is impossible to look at this as an enhancement, as is a major issue for one of the key flows for Terraform when migrating already existing infrastructure. Not only that, but sensitive Terraform variables and sensitive environment variables behave differently (cc @pkolyvas, as you removed the bug label), which further consolidate the impression that this is not an enhancement. See below for a workaround using environment variables.

---

A workaround I found that is effective and simple: instead of using Terraform variables, use environment variables.
Downside: you are not relying on Terraform variables anymore.
Upside: TFC supports sensitive environment variables and they work pretty well across systems.

From TFC is possible to set an environment variable as sensitive, and the terraform CLI seems to properly override the value locally if the environment variable is present, as import does not complain about the sensitive variable with a remote backend anymore and actually succeeds.

So:

1. do not set your sensitive credentials as variables in TFC
2. set them as sensitive environment variables
3. when running import, set the environment variable (e.g. export, dotenvs, VAR=foo command)
4. profit 🎉

This workaround has worked reliably for me. I'm not particularly happy, as variables are more explicit and terraform reports when a variable is missing, while with env vars you only see the effect of the missing variable (e.g. "Authentication error").
I work around that limitation with more documentation which is okish.

The great benefit is that there are no code changes or state management involved.

Hope this helps and works for you too!

PS: note that as the author highlights, using TF_VAR_ environment variables for variables does not work.

[dannystaple]

Is there any movement on this from terraform? This is an annoying bug indeed. I've currently got static overrides in files in a pycharm dont-commit list - but that is a security disaster waiting to happen.

[andydkelly-ig]

Agree with other posters - this definitely seems a bug and leaves Enterprises with hacky workarounds. A response from Hashicorp on this would be greatly appreciated. Cheers guys.

[robinbowes]

I've just hit this - what a total PITA. We're a paying customer. I shall file a support ticket.

[brenthc]

Hello all,

Terraform CLI Support Engineer here. Thanks for reaching out, Robin. I'd like to share what we discussed here for the benefit of the community.

There are a few ways you can work around this situation. In my experience, the simplest is to change the way the provider is defined locally so that local credentials can be used, and the easiest way to do that is to make an override file.

For example, consider importing an EIP, where the credentials for the AWS provider are given using sensitive variables in TFC / TFE:

# main.tf
terraform {
  backend "remote" {
    organization = "your_org"
    workspaces {
      name = "sens_import"
    }
  }
}

variable "access_key" {}
variable "secret_key" {}
variable "region" {}
variable "token" {}

provider "aws" {
  access_key = var.access_key
  secret_key = var.secret_key
  token      = var.token
  region     = var.region
}

resource "aws_eip" "e" {}

Terraform import will fail, which is what is reported in this issue. To unobtrusively rewrite the provider block, create an override file:

# main_override.tf
provider "aws" {
  access_key = null
  secret_key = null
  token      = null
}

Because configurations from overrides are merged, it's necessary to be explicit about unsetting the arguments by using null. Now, in my environment I can use ~/.aws/credentials or the different AWS environment variables to have the import succeed. The warning about the sensitive variables will still appear, but so long as they are no longer required for the import the import will succeed.

This issue has high visibility internally and addressing it is a part of a larger effort. The import operation is no small part, and so is marked as a blocker for that project. I'm afraid I have no timelines or release versions to share, but, again, please be assured that we are actively working on this.

Brent W.
Sr. Support Engineer, Terraform CLI

[robinbowes]

Heh, you beat me to it, @brenthc

I can confirm that this approach worked for me. In my case, I am using the TFE provider, and have the following provider declaration:

provider "tfe" {
  token = var.tfe_auth_token
}

I added an override file containing the following:

provider "tfe" {
  token = null
}

This allowed me to import the resources I wanted to import.

Thanks again, Brent.

[adnathanail]

Hi just wondering if there was any update on this? Quite annoying to have to override this as I have about 20 sensitive variables!

[m0ar]

Any update @brenthc ? This bug is a big nuisance in our move to Terraform :/

[phred-unity]

My team is also experiencing the same issue, any news on this?

[nikolay]

@phred-unity Are you on v1.1? It seems that things have changed since v1, but I didn't get a chance to test if the issue still exists in v1.1.

[vinaynb-at]

yes the issue still exists in v1.1, tested on debian amd64. @nikolay

[roccolangeweg-old]

Is there any ETA on this? This really convolutes the config that we cannot just override the variabel when running import locally.

[crw]

Thanks for adding your v1.1 experience to this issue.

The team is investigating, however there is no targeted release version for a change at this time. When that changes, we'll let you know on this issue thread. Thanks!

[vinaynb-at]

Okay seems something went wrong when testing with v1.1, this issue no longer exists on Terraform v1.1.3 on linux_amd64 cc @crw @nikolay. The below command works fine
terraform import -var-file="secret.tfvars" <aws_resource_name>.<resource_name> <arn>

[philippeboyd]

the issue still remains with v1.1.6

[nikolay]

@philippeboyd I haven't tried it as we, but this is saddening!

[jbg]

Due to this bug we migrated away from Terraform Cloud, would recommend that others do the same. It's one thing to kneecap your product like this, but it's another thing to ignore the feedback for 1.5 years.

[glenngillen]

There's a few things I should clarify about this as there's multiple issues being conflated. The good news is, I think they're all solved!

1. Overriding variables on the CLI when using Terraform Cloud or Terraform Enterprise: This is resolved in 1.1, but also requires you switch to using the new cloud integration for Terraform. This is not the cloud remote backend you may have previously been using. It came out with 1.1 and solving this issue was one of the large motivators for the new integration. If you make both of those changes I expect it will work as expected. I've recorded a demo to show how I can use CLI vars to override remove variables. I also tested toggling the sensitive state on that variable just to make doubly sure. Hopefully the video fills in any of the gaps in how to do this. Let me know if it does, as it will help work out what documentation we need to improve.
2. Providing variables to enable import: This is mostly the same as the above, but a more specialised case. So upgrade to 1.1 and switch to the new cloud integration. Again I've just run a test of importing a new IAM role and it's completed successfully. The things to be aware of is that if you're providing the values via -var you'll need to explicitly configure the provider to use those variables. The implicitly applied variables are normally environment variables, not Terraform variables. If you are using env vars then something like AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=xxxx terraform import resource identifier should work. I've tested that flow too just now and it succeeded. Unfortunately no video demos of that one as I don't have time to edit the video to redact my credentials.

Hopefully that helps.

[case]

@crw @brenthc I'm running into this issue as well, currently using Terraform v1.1.9. Curious if there are any updates from your side? I've upgraded to the cloud {} config that was introduced in 1.1 (instead of the older remote {} config) but this issue is still occurring.

(e.g. The values I have in secrets.auto.tfvars are being ignored, the null values override.tf approach mentioned above is triggering errors saying "The argument {my_key} is required, but no definition was found.")

Updated:

Oops, according to this Changelog entry, this fix is coming in 1.2.0?

[crw]

@case That is my understanding as well, based on the comments in the associated PR (#29972).

Although 1.2 is not yet at a production-ready release state, you should be able to experiment with this in the latest beta release. See: https://releases.hashicorp.com/terraform/

[danielmahon]

Using the secret cloud env as the fallback, allowed local imports to work for me (gcp.key is not in the repo)
still not ideal though

provider "google" {
  project     = local.project_id
  region      = local.project_region
  zone        = local.project_zone
  credentials = fileexists("gcp.key") ? "gcp.key" : var.GOOGLE_CREDENTIALS
}

[emmeowzing]

Can confirm upgrading my tf version to 1.2.5 (latest at this time) fixed this error.

[dkirrane]

Just to clarify to get terraform import to work with Terraform Cloud I'd need to create a temporary secrets.auto.tfvars locally and add all the sensitive variables from my Terraform Cloud workspace to it?
Is there any plan to actually run terraform import remotely on Terraform Cloud to avoid this?