terraform-provider-google_v5.17.0_x5 plugin crashed!

[jay-kinder]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

1.7.2

Affected Resource(s)

google_monitoring_dashboard

Terraform Configuration

resource "google_monitoring_dashboard" "dashboard" {
  for_each = { for k, v in var.dashboards : k => v
  if length(var.dashboards) > 0 }

  project = var.project_id
  dashboard_json = jsonencode({
    "displayName" : "${each.key}",
    "labels" : { "managed_by_terraform" : "" },
    "${each.value.layout}Layout" : {
      "columns" : each.value.columns,
      "widgets" : [
        concat([for widget in each.value.enabled_widgets :
          jsondecode(file("${path.module}/widgets/${widget}.json"))
      ])]
    }
  })
}

Debug Output

Stack trace from the terraform-provider-google_v5.17.0_x5 plugin:

panic: interface conversion: interface {} is []interface {}, not map[string]interface {}

goroutine 360 [running]:
github.com/hashicorp/terraform-provider-google/google/services/monitoring.removeComputedKeys(0x2cfeda0?, 0x400103b0e0?)
        github.com/hashicorp/terraform-provider-google/google/services/monitoring/resource_monitoring_dashboard.go:37 +0x578
github.com/hashicorp/terraform-provider-google/google/services/monitoring.removeComputedKeys(0x4001e13b00?, 0x8cf?)
        github.com/hashicorp/terraform-provider-google/google/services/monitoring/resource_monitoring_dashboard.go:30 +0x270
github.com/hashicorp/terraform-provider-google/google/services/monitoring.monitoringDashboardDiffSuppress({0x39e03f8?, 0x32e7866?}, {0x4001e06900, 0x8c3}, {0x4001e12900, 0x8cf}, 0x40012e3080?)
        github.com/hashicorp/terraform-provider-google/google/services/monitoring/resource_monitoring_dashboard.go:57 +0xc0
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.diff(0x4001acae68?, {0x39ccea8, 0x400102f8f0}, {0x32e7866, 0xe}, 0x4000fb9400, 0x40012e2f00, {0x39d2de0?, 0x40012e3080}, 0x0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/schema.go:1144 +0x268
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0x4000faa810, {0x39ccea8, 0x400102f8f0}, 0x40018fda00, 0x400103a510, 0x4000f8a240, {0x32a33c0, 0x4000435500}, 0x0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/schema.go:679 +0x2a4
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0x39cdce0?, {0x39ccea8?, 0x400102f8f0?}, 0x40018fda00, 0x2cfeda0?, {0x32a33c0?, 0x4000435500?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource.go:890 +0x50
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0x4000539c38, {0x39ccea8?, 0x400102f7d0?}, 0x4001107450)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/grpc_provider.go:741 +0x810
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.PlanResourceChange({0x4001005320, 0x4001005380, {0x40018c04a0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_PlanResourceChange.go:27 +0xdc
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0x4000566960, {0x39ccea8?, 0x4000f79f20?}, 0x400030a930)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:783 +0x3b8
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0x3215900?, 0x4000566960}, {0x39ccea8, 0x4000f79f20}, 0x4002325e80, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:367 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x40016cb600, {0x39ccea8, 0x4000f79e90}, {0x39d6858, 0x40019aa1a0}, 0x400163a6c0, 0x400199e060, 0x52a4de8, 0x0)
        google.golang.org/grpc@v1.60.1/server.go:1372 +0xbec
google.golang.org/grpc.(*Server).handleStream(0x40016cb600, {0x39d6858, 0x40019aa1a0}, 0x400163a6c0)
        google.golang.org/grpc@v1.60.1/server.go:1783 +0xc7c
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        google.golang.org/grpc@v1.60.1/server.go:1016 +0x5c
created by google.golang.org/grpc.(*Server).serveStreams.func2
        google.golang.org/grpc@v1.60.1/server.go:1027 +0x144

Error: The terraform-provider-google_v5.17.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Expected Behavior

To read the terraform resource and show no changes (or a change in place, if required)

Actual Behavior

When first running this code, it will work perfectly and add the dashboard with the right widgets.

However, any following plans (unless it causes a recreation) will cause a provider crash.

I have tried 5.18 and 5.16 provider also, and received the same crash.

Steps to reproduce

1. terraform plan (after initial creation)

Important Factoids

No response

References

Example Widget:

{
    "title": "CPU Utilisation",
    "scorecard": {
        "gaugeView": {
            "lowerBound": 0,
            "upperBound": 100
        },
        "thresholds": [
            {
                "color": "RED",
                "direction": "ABOVE",
                "label": "",
                "value": 95
            },
            {
                "color": "YELLOW",
                "direction": "ABOVE",
                "label": "",
                "value": 90
            },
            {
                "color": "YELLOW",
                "direction": "BELOW",
                "label": "",
                "value": 0
            }
        ],
        "timeSeriesQuery": {
            "timeSeriesQueryLanguage": "{ t_0:\n   fetch k8s_node\n   | metric 'kubernetes.io/node/cpu/core_usage_time'\n| ${project_id}\n| ${location}\n| ${cluster_name}\n   | align rate(1m)\n   | every 1m\n   | group_by [], [value_core_usage_time_aggregate: aggregate(value.core_usage_time)]; \n   t_1:\nfetch k8s_node\n| metric 'kubernetes.io/node/cpu/total_cores'\n| ${project_id}\n| ${location}\n| ${cluster_name}\n| group_by 1m, [value_total_cores_mean: mean(value.total_cores)]\n| every 1m\n| group_by [], [value_total_cores_mean_aggregate: sum(value_total_cores_mean)]}\n| join\n| window 5m\n| value\n   [v_0:\n      cast_units(\n        div(t_0.value_core_usage_time_aggregate,\n          t_1.value_total_cores_mean_aggregate) * 100,\n        \"%\")]",
            "unitOverride": ""
        }
    }
}

b/327438769

[zli82016]

It looks like a bug in the provider code and will be forwarded to the monitoring service team to fix.

[zli82016]

@jay-kinder , can you please provide the result for terraform state show google_monitoring_dashboard.dashboard? Thanks.

[jay-kinder]

Command:

tf state show 'module.dashboards.module.dashboards.google_monitoring_dashboard.dashboard["Default SRE Dashboard"]'

Output:

# module.dashboards.module.dashboards.google_monitoring_dashboard.dashboard["Default SRE Dashboard"]:
resource "google_monitoring_dashboard" "dashboard" {
    dashboard_json = jsonencode(
        {
            displayName = "Default SRE Dashboard"
            etag        = "03b9b9542fef7ea215cfea91c74df95d"
            gridLayout  = {
                columns = "2"
                widgets = [
                    {
                        scorecard = {
                            gaugeView       = {
                                upperBound = 100
                            }
                            thresholds      = [
                                {
                                    color     = "RED"
                                    direction = "ABOVE"
                                    value     = 95
                                },
                                {
                                    color     = "YELLOW"
                                    direction = "ABOVE"
                                    value     = 90
                                },
                                {
                                    color     = "YELLOW"
                                    direction = "BELOW"
                                },
                            ]
                            timeSeriesQuery = {
                                timeSeriesQueryLanguage = <<-EOT
                                    { t_0:
                                       fetch k8s_node
                                       | metric 'kubernetes.io/node/cpu/core_usage_time'
                                    | ${project_id}
                                    | ${location}
                                    | ${cluster_name}
                                       | align rate(1m)
                                       | every 1m
                                       | group_by [], [value_core_usage_time_aggregate: aggregate(value.core_usage_time)]; 
                                       t_1:
                                    fetch k8s_node
                                    | metric 'kubernetes.io/node/cpu/total_cores'
                                    | ${project_id}
                                    | ${location}
                                    | ${cluster_name}
                                    | group_by 1m, [value_total_cores_mean: mean(value.total_cores)]
                                    | every 1m
                                    | group_by [], [value_total_cores_mean_aggregate: sum(value_total_cores_mean)]}
                                    | join
                                    | window 5m
                                    | value
                                       [v_0:
                                          cast_units(
                                            div(t_0.value_core_usage_time_aggregate,
                                              t_1.value_total_cores_mean_aggregate) * 100,
                                            "%")]
                                EOT
                            }
                        }
                        title     = "CPU Utilisation"
                    },
                    {
                        scorecard = {
                            gaugeView       = {
                                upperBound = 100
                            }
                            thresholds      = [
                                {
                                    color     = "RED"
                                    direction = "ABOVE"
                                    value     = 95
                                },
                                {
                                    color     = "YELLOW"
                                    direction = "ABOVE"
                                    value     = 90
                                },
                            ]
                            timeSeriesQuery = {
                                timeSeriesQueryLanguage = <<-EOT
                                    { t_0:
                                    fetch k8s_node
                                    | metric 'kubernetes.io/node/memory/used_bytes'
                                    | ${project_id}
                                    | ${location}
                                    | ${cluster_name}
                                    | group_by 1m, [value_used_bytes_mean: mean(value.used_bytes)]
                                    | every 1m
                                    | group_by [],
                                        [value_used_bytes_mean_aggregate: aggregate(value_used_bytes_mean)]
                                    
                                       ; 
                                       t_1:
                                    fetch k8s_node
                                    | metric 'kubernetes.io/node/memory/total_bytes'
                                    | ${project_id}
                                    | ${location}
                                    | ${cluster_name}
                                    | group_by 1m, [value_total_bytes_mean: mean(value.total_bytes)]
                                    | every 1m
                                    | group_by [],
                                        [value_total_bytes_mean_aggregate: aggregate(value_total_bytes_mean)]}
                                    | join
                                    | window 5m
                                    | value
                                       [v_0:
                                          cast_units(
                                            div(t_0.value_used_bytes_mean_aggregate,
                                              t_1.value_total_bytes_mean_aggregate) * 100,
                                            "%")]
                                EOT
                            }
                        }
                        title     = "Memory Utilisation"
                    },
                ]
            }
            labels      = {
                managed_by_terraform = ""
            }
            name        = "projects/610588521293/dashboards/58e8b192-fb09-4979-a162-4cf8ebea9e88"
        }
    )
    id             = "projects/610588521293/dashboards/58e8b192-fb09-4979-a162-4cf8ebea9e88"
    project        = "sre-central-monitoring"
}

[zli82016]

Thanks, @jay-kinder . Before running the second terraform plan, have you updated the resource configuration?

I tried to reproduce the issue with the resource state you provided and didn't see the provider crash error.

resource "google_monitoring_dashboard" "dashboard" {
    dashboard_json = jsonencode(
        {
            displayName = "Default SRE Dashboard"
            gridLayout  = {
                columns = "2"
                widgets = [
                    {
                        scorecard = {
                            gaugeView       = {
                                upperBound = 100
                            }
                            thresholds      = [
                                {
                                    color     = "RED"
                                    direction = "ABOVE"
                                    value     = 95
                                },
                                {
                                    color     = "YELLOW"
                                    direction = "ABOVE"
                                    value     = 90
                                },
                                {
                                    color     = "YELLOW"
                                    direction = "BELOW"
                                },
                            ]
                            timeSeriesQuery = {
                                timeSeriesQueryLanguage = <<-EOT
                                    { t_0:
                                       fetch k8s_node
                                       | metric 'kubernetes.io/node/cpu/core_usage_time'
                                    | "terraform-dev-zhenhuali"
                                       | align rate(1m)
                                       | every 1m
                                       | group_by [], [value_core_usage_time_aggregate: aggregate(value.core_usage_time)];
                                       t_1:
                                    fetch k8s_node
                                    | metric 'kubernetes.io/node/cpu/total_cores'
                                    | group_by 1m, [value_total_cores_mean: mean(value.total_cores)]
                                    | every 1m
                                    | group_by [], [value_total_cores_mean_aggregate: sum(value_total_cores_mean)]}
                                    | join
                                    | window 5m
                                    | value
                                       [v_0:
                                          cast_units(
                                            div(t_0.value_core_usage_time_aggregate,
                                              t_1.value_total_cores_mean_aggregate) * 100,
                                            "%")]
                                EOT
                            }
                        }
                        title     = "CPU Utilisation"
                    },
                    {
                        scorecard = {
                            gaugeView       = {
                                upperBound = 100
                            }
                            thresholds      = [
                                {
                                    color     = "RED"
                                    direction = "ABOVE"
                                    value     = 95
                                },
                                {
                                    color     = "YELLOW"
                                    direction = "ABOVE"
                                    value     = 90
                                },
                            ]
                            timeSeriesQuery = {
                                timeSeriesQueryLanguage = <<-EOT
                                    { t_0:
                                    fetch k8s_node
                                    | metric 'kubernetes.io/node/memory/used_bytes'
                                    | group_by 1m, [value_used_bytes_mean: mean(value.used_bytes)]
                                    | every 1m
                                    | group_by [],
                                        [value_used_bytes_mean_aggregate: aggregate(value_used_bytes_mean)]

                                       ;
                                       t_1:
                                    fetch k8s_node
                                    | metric 'kubernetes.io/node/memory/total_bytes'
                                    | group_by 1m, [value_total_bytes_mean: mean(value.total_bytes)]
                                    | every 1m
                                    | group_by [],
                                        [value_total_bytes_mean_aggregate: aggregate(value_total_bytes_mean)]}
                                    | join
                                    | window 5m
                                    | value
                                       [v_0:
                                          cast_units(
                                            div(t_0.value_used_bytes_mean_aggregate,
                                              t_1.value_total_bytes_mean_aggregate) * 100,
                                            "%")]
                                EOT
                            }
                        }
                        title     = "Memory Utilisation"
                    },
                ]
            }
            labels      = {
                managed_by_terraform = ""
            }
        }
    )
}

[jay-kinder]

Hello

If I run it with no changes then it will work as expected

However, if I add another widget through the enabled_widgets variable, I will receive the provider crash. It seems to only happen if I want to make changes after the initial deployment.

Thanks

[zli82016]

@jay-kinder, thanks for the information. I was trying to add a new widget, and then terraform apply succeeded. I cannot reproduce this issue. I wonder if I missed something to reproduce it. I didn't use module.

[jay-kinder]

Hi

No probs thanks for looking into it...I can give you some more info to try to reproduce:

external module main.tf:

resource "google_monitoring_dashboard" "dashboard" {
  for_each = { for k, v in var.dashboards : k => v
  if length(var.dashboards) > 0 }

  project = var.project_id
  dashboard_json = jsonencode({
    "displayName" : "${each.key}",
    "labels" : { "managed_by_terraform" : "" },
    "${each.value.layout}Layout" : {
      "columns" : each.value.columns,
      "widgets" : [
        concat([for widget in each.value.enabled_widgets :
          jsondecode(file("${path.module}/widgets/${widget}.json"))
      ])]
    }
  })
}

external module vars:

variable "project_id" { description = "The project to create resources in" }
variable "dashboards" {
  description = "Map of dashboards with widgets enabled"
  type = map(object({
    enabled_widgets = list(string)
    layout          = optional(string, "grid")
    columns         = optional(string, "2")
  }))
  default = {}
}

external module example widget:

{
    "title": "VM CPU Utilisation",
    "xyChart": {
        "chartOptions": {
            "mode": "COLOR"
        },
        "dataSets": [
            {
                "minAlignmentPeriod": "60s",
                "plotType": "LINE",
                "targetAxis": "Y1",
                "timeSeriesQuery": {
                    "timeSeriesFilter": {
                        "aggregation": {
                            "alignmentPeriod": "60s",
                            "crossSeriesReducer": "REDUCE_MEAN",
                            "groupByFields": [],
                            "perSeriesAligner": "ALIGN_MEAN"
                        },
                        "filter": "metric.type=\"compute.googleapis.com/instance/cpu/utilization\" resource.type=\"gce_instance\""
                    }
                }
            }
        ],
        "thresholds": [],
        "yAxis": {
            "label": "",
            "scale": "LINEAR"
        }
    }
}

module call:

module "dashboards" {
  # source = "../../infra-modules/dashboards" # used for local testing
  source = "git::ssh://git@github.com/Cloud-Technology-Solutions/sre-central-monitoring-modules//dashboards/"

  project_id = var.project

  dashboards = {
    "Default SRE Dashboard" = {
      enabled_widgets = [
        "cpu_util",
        "mem_util"
      ]
    }
  }
}

main.tf:

module "dashboards" {
  source = "./dashboards"

  project = local.project
}

hope that helps!

[zli82016]

Hello, @jay-kinder , thank you for providing the detailed configuration. I can reproduce it.

[zli82016]

The issue is that the syntax is wrong when concat widgets. concat is not needed here.
It should be

      "widgets" : [for widget in each.value.enabled_widgets :
          jsondecode(file("${path.module}/widgets/${widget}.json"))
      ]

Resource config

resource "google_monitoring_dashboard" "dashboard" {
  for_each = { for k, v in var.dashboards : k => v
  if length(var.dashboards) > 0 }

  project = var.project_id
  dashboard_json = jsonencode({
    "displayName" : "${each.key}",
    "labels" : { "managed_by_terraform" : "" },
    "${each.value.layout}Layout" : {
      "columns" : each.value.columns,
      "widgets" : [for widget in each.value.enabled_widgets :
          jsondecode(file("${path.module}/widgets/${widget}.json"))
      ]
    }
  })
}

concat doc
https://developer.hashicorp.com/terraform/language/functions/concat