1.13.1: Invalid resource instance data in state

[jkroepke]

After upgrading from 1.12 to 1.13.1, I have multiple plan issues:

│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 27:
│   27: resource "azapi_resource" "gallery_application_version_linux" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_linux["1.1.2"]
│ data could not be decoded from the state: missing expected {.

[jkroepke]

Seems like a breaking change in 1.13. Body now requires a real HCL structure instead JSON

[jkroepke]

The error persists even after migrate the code.

[ms-henglu]

Hi @jkroepke ,

Thank you for taking time to open this issue.

The v1.13.x introduces the dynamic properties which allow user to use a real HCL, but the old JSON solution still works, so it's not required to migrate.

The error message comes from the terraform cli: https://github.com/hashicorp/terraform/blob/main/internal/terraform/evaluate.go#L742 and it states "this should not happen" 🤔

Would you please share the terraform version that you're using? And would you try the latest terraform?

And I have a workaround that might fix it:

1. Downgrade to v1.12.1 and make sure the terraform state is valid
2. Upgrade to v1.13.0, and run terraform plan command, it's expected to see there's no changes.
3. Run terraform refresh command to refresh the terraform state.

[jkroepke]

I just upgrade from 1.12.1 to 1.13.1 and it fails at step 2.

I got this error in both situation, where I don't touch the code and where I apply the migration to dynamic HCL.

Maybe the error is gallery application version specific?

Downgrade to 1.12.1, plan and apply works fine.

[ms-henglu]

Thanks for trying the workaround.

Maybe the error is gallery application version specific?

Yes, it seems so.

Would you please share me the config to reproduce this issue? Thanks

[jkroepke]

the resource is extremely integrated into our module.

Maybe this is sufficient.

resource "azapi_resource" "gallery_application_version_windows" {
  for_each = toset(var.installer_versions)

  type      = "Microsoft.Compute/galleries/applications/versions@2022-03-03"
  name      = each.key
  parent_id = azurerm_gallery_application.windows.id
  location  = azurerm_gallery_application.windows.location

  locks = [azurerm_gallery_application.windows.id]
  tags  = var.tags

  body = nonsensitive(jsonencode({
    properties = {
      publishingProfile = {
        enableHealthCheck = false
        customActions = [
          {
            name        = "opsstack-agent-reinstall"
            description = "Running the installer again"
            script      = "cmd.exe /c .\\${azurerm_gallery_application.windows.name}.cmd"
          }
        ]

        manageActions = {
          install = "cmd.exe /c .\\${azurerm_gallery_application.windows.name}.cmd"
          remove  = "ls"
          update  = "cmd.exe /c .\\${azurerm_gallery_application.windows.name}.cmd"
        }

        replicationMode = "Full"
        replicaCount    = 1
        settings = {
          packageFileName = "${azurerm_gallery_application.windows.name}.cmd"
        }
        source = {
          mediaLink = "${azurerm_storage_blob.windows[each.key].id}${data.azurerm_storage_account_blob_container_sas.this.sas}"
        }
        storageAccountType = "Standard_ZRS"
        targetRegions = flatten([
          for _, region in concat(
            [
              {
                region   = azurerm_gallery_application.windows.location
                replicas = 1
              }
            ], var.opsstack_agent_additional_regions
          ) :
          [
            {
              name                 = local.location_cli_mapping[region.region]
              regionalReplicaCount = region.replicas
              storageAccountType   = "Standard_ZRS"
            }
          ]
        ]),
      },
      safetyProfile = {
        allowDeletionOfReplicatedLocations = true
      }
    }
  }))

  ignore_body_changes = [
    "properties.publishingProfile.source.media_link"
  ]

  response_export_values = ["properties.provisioningState"]

  lifecycle {
    replace_triggered_by = [
      azurerm_storage_blob.linux[each.key].id
    ]

    postcondition {
      condition     = jsondecode(self.output).properties.provisioningState == "Succeeded"
      error_message = "${self.name} returned an provisioningState != ${jsondecode(self.output).properties.provisioningState}"
    }
  }
}

[ms-henglu]

Thanks for the config, I tried to use this config to reproduce it, but failed. Here's my complete config which is created by v1.12.1 and is able to upgrade to v1.13.0. And I'm using the latest Terraform version.

Hope it could help you.

resource "azurerm_resource_group" "test" {
  name     = "acctest-compute-henglu"
  location = "westeurope"
}

resource "azurerm_shared_image_gallery" "test" {
  name                = "henglu429"
  resource_group_name = azurerm_resource_group.test.name
  location            = azurerm_resource_group.test.location
}

resource "azurerm_gallery_application" "test" {
  name              = "henglu429"
  gallery_id        = azurerm_shared_image_gallery.test.id
  location          = azurerm_resource_group.test.location
  supported_os_type = "Windows"
}

resource "azurerm_storage_account" "test" {
  name                     = "henglu429"
  resource_group_name      = azurerm_resource_group.test.name
  location                 = azurerm_resource_group.test.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "test" {
  name                  = "henglu429"
  storage_account_name  = azurerm_storage_account.test.name
  container_access_type = "blob"
}

resource "azurerm_storage_blob" "test" {
  name                   = "scripts"
  storage_account_name   = azurerm_storage_account.test.name
  storage_container_name = azurerm_storage_container.test.name
  type                   = "Page" # Use Page Blob as a workaround to UnmanagedStorageAccount quota issue
  size                   = 512
}

data "azurerm_storage_account_blob_container_sas" "test" {
  connection_string = "${azurerm_storage_account.test.primary_connection_string}"
  container_name    = "${azurerm_storage_container.test.name}"
  https_only        = true

  start  = "2017-03-21"
  expiry = "2022-03-21"

  permissions {
    read   = true
    add    = false
    create = false
    write  = true
    delete = false
    list   = true
  }
}

resource "azapi_resource" "gallery_application_version_windows" {
  type      = "Microsoft.Compute/galleries/applications/versions@2022-03-03"
  name      = "0.0.1"
  parent_id = azurerm_gallery_application.test.id
  location  = azurerm_gallery_application.test.location

  locks = [azurerm_gallery_application.test.id]
  body = nonsensitive(jsonencode({
    properties = {
      publishingProfile = {
        enableHealthCheck = false
        customActions = [
          {
            name        = "opsstack-agent-reinstall"
            description = "Running the installer again"
            script      = "cmd.exe /c .\\${azurerm_gallery_application.test.name}.cmd"
          }
        ]

        manageActions = {
          install = "cmd.exe /c .\\${azurerm_gallery_application.test.name}.cmd"
          remove  = "ls"
          update  = "cmd.exe /c .\\${azurerm_gallery_application.test.name}.cmd"
        }

        replicationMode = "Full"
        replicaCount    = 1
        settings = {
          packageFileName = "${azurerm_gallery_application.test.name}.cmd"
        }
        source = {
          mediaLink = azurerm_storage_blob.test.id
        }
        storageAccountType = "Standard_ZRS"
        targetRegions = [
            {
              name                 = "West Europe"
              regionalReplicaCount = 1
              storageAccountType   = "Standard_ZRS"
            }
          ]
      },
      safetyProfile = {
        allowDeletionOfReplicatedLocations = true
      }
    }
  }))

  ignore_body_changes = [
    "properties.publishingProfile.source.media_link"
  ]

  response_export_values = ["properties.provisioningState"]

}

[ms-henglu]

Is it possible to remove them from the state and import them by v1.13.0? Though I guess it might not.

[jkroepke]

The only different I see is that azurerm_storage_account_blob_container_sas is not used on the mediaLink properties and I used for_each to generated multiple. But I guess it wont change anything.

I mean almost two days of work to do that. Having 13 VM applications across 15 customers deployments. Based on how VM Applications are working, they can't be delete in parallel and it takes approx. 5 up to 10 minutes per version to remove.

Its also mentionable that I got the error for the same resource multiple times:

╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.3"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.3"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.2"]
│ data could not be decoded from the state: missing expected {.
╵
╷
│ Error: Invalid resource instance data in state
│ 
│   on ../../terraform/modules/azure_opsstack/agent_gallery_application.tf line 106:
│  106: resource "azapi_resource" "gallery_application_version_windows" {
│ 
│ Instance
│ module.opsstack.azapi_resource.gallery_application_version_windows["1.1.3"]
│ data could not be decoded from the state: missing expected {.

If you want, you can share your state from your test (after apply 1.12) and I can compare your state with my state and look for differences.

[ms-henglu]

Thanks for the details!

And I'd like to double confirm your terraform version. Recently I saw some issues are resolved by the latest terraform version.

Here're the states for v1.12.1 and v1.13.1. I've redacted all the sensitive data. Hope it could help.

v1.12.1:

    {
      "mode": "managed",
      "type": "azapi_resource",
      "name": "gallery_application_version_windows",
      "provider": "provider[\"registry.terraform.io/azure/azapi\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "body": "{\"properties\":{\"publishingProfile\":{\"customActions\":[{\"description\":\"Running the installer again\",\"name\":\"opsstack-agent-reinstall\",\"script\":\"cmd.exe /c .\\\\example.cmd\"}],\"enableHealthCheck\":false,\"manageActions\":{\"install\":\"cmd.exe /c .\\\\example.cmd\",\"remove\":\"ls\",\"update\":\"cmd.exe /c .\\\\example.cmd\"},\"replicaCount\":1,\"replicationMode\":\"Full\",\"settings\":{\"packageFileName\":\"example.cmd\"},\"source\":{\"mediaLink\":\"https://example.blob.core.windows.net/example/scripts\"},\"storageAccountType\":\"Standard_ZRS\",\"targetRegions\":[{\"name\":\"West Europe\",\"regionalReplicaCount\":1,\"storageAccountType\":\"Standard_ZRS\"}]},\"safetyProfile\":{\"allowDeletionOfReplicatedLocations\":true}}}",
            "id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example/versions/0.0.1",
            "identity": [],
            "ignore_body_changes": [
              "properties.publishingProfile.source.media_link"
            ],
            "ignore_casing": false,
            "ignore_missing_property": true,
            "location": "westeurope",
            "locks": [
              "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example"
            ],
            "name": "0.0.1",
            "output": "{\"properties\":{\"provisioningState\":\"Succeeded\"}}",
            "parent_id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example",
            "removing_special_chars": false,
            "response_export_values": [
              "properties.provisioningState"
            ],
            "schema_validation_enabled": true,
            "tags": {},
            "timeouts": null,
            "type": "Microsoft.Compute/galleries/applications/versions@2022-03-03"
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJkZWxldGUiOjE4MDAwMDAwMDAwMDAsInJlYWQiOjMwMDAwMDAwMDAwMCwidXBkYXRlIjoxODAwMDAwMDAwMDAwfX0=",
          "dependencies": [
            "azurerm_gallery_application.test",
            "azurerm_resource_group.test",
            "azurerm_shared_image_gallery.test",
            "azurerm_storage_account.test",
            "azurerm_storage_blob.test",
            "azurerm_storage_container.test"
          ]
        }
      ]
    }

v1.13.1:

    {
      "mode": "managed",
      "type": "azapi_resource",
      "name": "gallery_application_version_windows",
      "provider": "provider[\"registry.terraform.io/azure/azapi\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "body": {
              "value": "{\"properties\":{\"publishingProfile\":{\"customActions\":[{\"description\":\"Running the installer again\",\"name\":\"opsstack-agent-reinstall\",\"script\":\"cmd.exe /c .\\\\example.cmd\"}],\"enableHealthCheck\":false,\"manageActions\":{\"install\":\"cmd.exe /c .\\\\example.cmd\",\"remove\":\"ls\",\"update\":\"cmd.exe /c .\\\\example.cmd\"},\"replicaCount\":1,\"replicationMode\":\"Full\",\"settings\":{\"packageFileName\":\"example.cmd\"},\"source\":{\"mediaLink\":\"https://example.blob.core.windows.net/example/scripts\"},\"storageAccountType\":\"Standard_ZRS\",\"targetRegions\":[{\"name\":\"West Europe\",\"regionalReplicaCount\":1,\"storageAccountType\":\"Standard_ZRS\"}]},\"safetyProfile\":{\"allowDeletionOfReplicatedLocations\":true}}}",
              "type": "string"
            },
            "id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example/versions/0.0.1",
            "identity": [],
            "ignore_body_changes": [
              "properties.publishingProfile.source.media_link"
            ],
            "ignore_casing": false,
            "ignore_missing_property": true,
            "location": "westeurope",
            "locks": [
              "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example"
            ],
            "name": "0.0.1",
            "output": {
              "value": "{\"properties\":{\"provisioningState\":\"Succeeded\"}}",
              "type": "string"
            },
            "parent_id": "/subscriptions/subscription_id/resourceGroups/acctest-compute-henglu/providers/Microsoft.Compute/galleries/example/applications/example",
            "removing_special_chars": false,
            "response_export_values": [
              "properties.provisioningState"
            ],
            "schema_validation_enabled": true,
            "tags": {},
            "timeouts": null,
            "type": "Microsoft.Compute/galleries/applications/versions@2022-03-03"
          },
          "sensitive_attributes": [],
          "dependencies": [
            "azurerm_gallery_application.test",
            "azurerm_resource_group.test",
            "azurerm_shared_image_gallery.test",
            "azurerm_storage_account.test",
            "azurerm_storage_blob.test",
            "azurerm_storage_container.test"
          ]
        }
      ]
    }

[jkroepke]

TF version: v1.8.2

I also noice that other azapi resource are note affected. At least I do not see any errors. But Terraform shows only 100 errors and the first 100 errors coming from the azapi gallery_application_version

[jkroepke]

Thats the state on my side:

{
      "module": "module.opsstack",
      "mode": "managed",
      "type": "azapi_resource",
      "name": "gallery_application_version_windows",
      "provider": "provider[\"registry.terraform.io/azure/azapi\"]",
      "instances": [
        {
          "index_key": "1.1.0",
          "schema_version": 0,
          "attributes": {
            "body": "{\"properties\":{\"publishingProfile\":{\"customActions\":[{\"description\":\"Running the Ops.Stack Agent installer again\",\"name\":\"opsstack-agent-reinstall\",\"script\":\"cmd.exe /c .\\\\opsstack-agent-windows.cmd\"}],\"enableHealthCheck\":false,\"manageActions\":{\"install\":\"cmd.exe /c .\\\\opsstack-agent-windows.cmd\",\"remove\":\"ls\",\"update\":\"cmd.exe /c .\\\\opsstack-agent-windows.cmd\"},\"replicaCount\":1,\"replicationMode\":\"Full\",\"settings\":{\"packageFileName\":\"opsstack-agent-windows.cmd\"},\"source\":{\"mediaLink\":\"https://removed.blob.core.windows.net/opsstack-installer-scripts/windows/install1.1.0.cmd?sv=2018-11-09\\u0026sr=c\\u0026st=2020-01-01\\u0026se=2040-01-01\\u0026sp=rl\\u0026spr=https\\u0026sig=removed\"},\"storageAccountType\":\"Standard_ZRS\",\"targetRegions\":[{\"name\":\"West Europe\",\"regionalReplicaCount\":1,\"storageAccountType\":\"Standard_ZRS\"},{\"name\":\"East US\",\"regionalReplicaCount\":2,\"storageAccountType\":\"Standard_ZRS\"}]},\"safetyProfile\":{\"allowDeletionOfReplicatedLocations\":true}}}",
            "id": "/subscriptions/subscription_id/resourceGroups/rg-opsstack-westeurope-001/providers/Microsoft.Compute/galleries/GAL_OPSSTACK_WESTEUROPE_001/applications/opsstack-agent-windows/versions/1.1.0",
            "identity": [],
            "ignore_body_changes": [
              "properties.publishingProfile.source.media_link"
            ],
            "ignore_casing": false,
            "ignore_missing_property": true,
            "location": "westeurope",
            "locks": [
              "/subscriptions/subscription_id/resourceGroups/rg-opsstack-westeurope-001/providers/Microsoft.Compute/galleries/GAL_OPSSTACK_WESTEUROPE_001/applications/opsstack-agent-windows"
            ],
            "name": "1.1.0",
            "output": "{\"properties\":{\"provisioningState\":\"Succeeded\"}}",
            "parent_id": "/subscriptions/subscription_id/resourceGroups/rg-opsstack-westeurope-001/providers/Microsoft.Compute/galleries/GAL_OPSSTACK_WESTEUROPE_001/applications/opsstack-agent-windows",
            "removing_special_chars": false,
            "response_export_values": [
              "properties.provisioningState"
            ],
            "schema_validation_enabled": true,
            "tags": {
              "department": "opsstack",
              "owner": "cloudeteer",
              "provisioner": "Terraform"
            },
            "timeouts": null,
            "type": "Microsoft.Compute/galleries/applications/versions@2022-03-03"
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJkZWxldGUiOjE4MDAwMDAwMDAwMDAsInJlYWQiOjMwMDAwMDAwMDAwMCwidXBkYXRlIjoxODAwMDAwMDAwMDAwfX0=",
          "dependencies": [
            "module.opsstack.azurerm_gallery_application.windows",
            "module.opsstack.azurerm_public_ip.this",
            "module.opsstack.azurerm_resource_group.this",
            "module.opsstack.azurerm_shared_image_gallery.this",
            "module.opsstack.azurerm_storage_account.gallery",
            "module.opsstack.azurerm_storage_blob.linux",
            "module.opsstack.azurerm_storage_blob.windows",
            "module.opsstack.azurerm_storage_container.gallery_opsstack_installer_scripts",
            "module.opsstack.data.azurerm_resource_group.this",
            "module.opsstack.data.azurerm_storage_account_blob_container_sas.this",
            "module.opsstack.random_string.opsstack_gal_storage_account_name"
          ]
        },

Differents to observe:

• I'm using azapi inside a module
• I'm using tags
• I'm using the resources with for_each
• I have dependencies against data resources (using it at mediaLink)

It's noticeable that my mediaLink contains a lot of special chars, maybe this causes the issue.

[ms-henglu]

Thanks for providing this!

I think we need to inspect the state in v1.13.1, would you please help with this experiment?

1. Backup your terraform state which could be used to restore
2. Upgrade to v1.13.1
3. Run terraform refresh command, then the state will be migrated.
4. Run terraform plan command, but I guess you would get same error.
5. Please share the terraform state.
6. Restore the terraform state and the provider version.