Feature Request: Verbose option for plan

[gitwitheprogram]

Current Terraform Version

Terraform v0.14.4

Use-cases

Running terraform plan now uses the concise format, which is awesome. Thank you. However, while the output includes lines that inform the user that attributes are hidden, terraform provides no easy way to toggle that output back on.

I confess that it is unlikely that I would use the full plan output often, but it could be helpful for doing a sanity check before certain changes. (for instance, I'm currently working on new modules and importing resources into them, and being able to easily validate that the reason an attribute isn't listed in the plan is because it is already set correctly could be helpful) I'd imagine that this would generally be used a lot more by newer Terraform users than experienced ones.

Attempted Solutions

I could easily write a wrapper script that sets the TF_X_CONCISE_DIFF environment variable to 0, runs terraform plan, and then unsets the variable, but that seems kind of silly to me. Other alternatives to see the settings of the other attributes would include running a terraform state show or pull prior to the plan. Maybe this is what is intended by the Terraform developers?

Proposal

Add a -verbose option to plan to revert to a full plan output.

References

[jgrumboe]

@gitwitheprogram
Sorry, no real answer to this, but at least you don't need to write a wrapper. Just set the value as a prefix to the command in the same line TF_X_CONCISE_DIFF=0 terraform plan. This will set the environment variable just for this one shot.

[gitwitheprogram]

True. However, I wasn't really looking for a better kludge. My point was just that having the -verbose flag here seems like a pretty standard thing. No user is going to know that they can set that environment variable without reading the release notes, as it's not mentioned in the -help output or in the documentation here that I can see.

It seems that the intention was to not provide functionality for users to get the full output, and I was just trying to make a case for it potentially being useful sometimes.

[techdragon]

I recently had a situation where the full output was essential to determining that terraform would do the correct thing before apply.

  # module.<redacted>.aws_security_group.<redacted>-dbs must be replaced
-/+ resource "aws_security_group" "<redacted>-dbs" {
      ~ arn                    = "<redacted>" -> (known after apply)
      ~ description            = "<redacted>" -> (known after apply) # forces replacement
      ~ id                     = "sg-<redacted>" -> (known after apply)
      ~ ingress                = [
          - {
              - cidr_blocks      = [
                  - "10.10.96.0/19",
                  - "10.10.128.0/19",
                  - "10.10.160.0/19",
                ]
              - description      = "Microsoft SQL Server"
              - from_port        = 1433
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 1433
            },
            # (1 unchanged element hidden)
        ]
      ~ name                   = "<redacted>" -> (known after apply) # forces replacement
      + name_prefix            = (known after apply)
      ~ owner_id               = "<redacted>" -> (known after apply)
        tags                   = {
            "ManagementTool" = "Terraform"
        }
      ~ vpc_id                 = "vpc-<redacted>" -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }

Looks like its going to purge the ingress from the object for no reason. while with the full output...

  # module.<redacted>.aws_security_group.<redacted>-dbs must be replaced
-/+ resource "aws_security_group" "<redacted>-dbs" {
      ~ arn                    = "<redacted>" -> (known after apply)
      ~ description            = "<redacted>" -> (known after apply) # forces replacement
        egress                 = [
            {
                cidr_blocks      = [
                    "0.0.0.0/0",
                ]
                description      = ""
                from_port        = 0
                ipv6_cidr_blocks = []
                prefix_list_ids  = []
                protocol         = "-1"
                security_groups  = []
                self             = false
                to_port          = 0
            },
        ]
      ~ id                     = "sg-<redacted>" -> (known after apply)
      ~ ingress                = [
            {
                cidr_blocks      = (known after apply)
                description      = "Microsoft SQL Server"
                from_port        = 1433
                ipv6_cidr_blocks = []
                prefix_list_ids  = []
                protocol         = "tcp"
                security_groups  = []
                self             = false
                to_port          = 1433
            },
          - {
              - cidr_blocks      = [
                  - "10.10.96.0/19",
                  - "10.10.128.0/19",
                  - "10.10.160.0/19",
                ]
              - description      = "Microsoft SQL Server"
              - from_port        = 1433
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 1433
            },
        ]
      ~ name                   = "<redacted>" -> (known after apply) # forces replacement
      + name_prefix            = (known after apply)
      ~ owner_id               = "<redacted>" -> (known after apply)
        revoke_rules_on_delete = false
        tags                   = {
            "ManagementTool" = "Terraform"
        }
      ~ vpc_id                 = "vpc-<redacted>" -> (known after apply) # forces replacement
    }

Makes it clear that terraform doesn't know what the current CIDR block is and that obviously when it applies the state it will perform the value lookup, get the current state (which is what it incorrectly wants to remove), realise its mistake and do nothing. Effectively I have a "phantom update" and the concise output hid this fact from me making it appear that terraform was going to do the wrong thing with no way to confirm it would do the correct thing had I not been able to access the full output without the new concise diff being applied.

I don't care if it stays behind an environment variable, TF_X_CONCISE_DIFF is fine for me, but I understand the current plan is removing this variable and giving us no way to get the full information when performing basic terraform operations (requiring us to muck around with the full JSON output tree to get the full output) is downright distasteful. It screams "we know better than you" and if followed through with will be the first time I come across a decision by the terraform development team that is to me, completely indefensible.

[bomb-on]

It would also be great to add an option to expand n lines of context around the line of interest, e.g. TF_X_DIFF_LINES=2 would show 2 lines above and below the line which would like to be modified, regardless of being part of some block or not.

The use case I had was changing environmental variables for AWS ECS task, which was showing only env var values and not their names, while in the same time showing surrounding blocks for other env vars which were not changing and which were not relevant to this plan at all:

  # aws_ecs_task_definition.this must be replaced
-/+ resource "aws_ecs_task_definition" "this" {
      ~ arn                      = "full_arn" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [
              ~ {
                  ~ environment           = [
                        {
                            name  = "ENVIRONMENTAL_VAR_1"
                            value = "env var 1 value"
                        },
                      ~ {
                          ~ value = "env var 2 value" -> "modified env var 2 value"
                            # (1 unchanged element hidden)
                        },
                        {
                            name  = "ENVIRONMENTAL_VAR_3"
                            value = "evn var 3 value"
                        },
                      ~ {
                          ~ value = "env var 4 value" -> "modified env var 4 value"
                            # (1 unchanged element hidden)
                        },
                        {
                            name  = "ENVIRONMENTAL_VAR_5"
                            value = "env var 5 value"
                        },
                        # (6 unchanged elements hidden)
                    ]
                    # (15 unchanged elements hidden)
                },
            ]
        )
      ~ id                       = "component_id" -> (known after apply)
      ~ revision                 = 6 -> (known after apply)
        tags                     = {
            "environment" = "production"
            # other tags...
        }
        # (7 unchanged attributes hidden)
    }

[soapergem]

Just came across this today for the first time after upgrading to Terraform v0.15.4 and it seems to me that even the TF_X_CONCISE_DIFF env var might be deprecated, as it doesn't seem to do anything...

I just want to echo the concerns of others about not being able to see the full plan. Seeing the full plan is part of the development cycle in some cases. If I'm interacting with some new or convoluted cloud service, I'll sometimes end up creating the resources manually in the console, then crafting my best attempt at a Terraform codification, then import my manually created resources, and finally run a plan to see what I got wrong. This is especially useful as new services come out and when the examples and documentation aren't as fleshed out as they need to be. Concealing the plan just makes everything far more difficult in this case.

[alkar]

Although the concise plan has worked wonders in preserving my sanity since it was introduced, I recently realised while working with aws_cloudfront_distribution that there is a valid case for having a fully verbose plan when working with such resources that have complex internal structures.

Specifically changes made to this resource's ordered_cache_behavior blocks are hard to understand when the context does not include some identifying attribute such as the path_pattern. For example:

[ ... ]
      ~ ordered_cache_behavior {
            # (12 unchanged attributes hidden)

          ~ forwarded_values {
                # (3 unchanged attributes hidden)

              ~ cookies {
                  ~ forward           = "none" -> "all"
                    # (1 unchanged attribute hidden)
                }
            }
        }
[ ... ]

[burner1024]

We need either a more verbose plan, or smarter one, where TF would do a better job at distinguishing repeatable blocks. I think verbose option is simpler?

[chadcarlton]

Looking at the env var, Frankly I am suprised this was taken away.

I am now in a position where terraform says there is a change, but hiding the change, due to the succint diff changes.

What changed on these two resources? They have a change "~" but no sub attributes listed..

Am I safe to apply? At this point I want to revert back to a version of terraform that actually tells me whats going on.

``

  # azurerm_api_management_api.non_prod_dnaapi-person["uat"] has been changed
  ~ resource "azurerm_api_management_api" "non_prod_dnaapi-person" {
        id                    = "/subscriptions/bob/resourceGroups/rg-apim1/providers/Microsoft.ApiManagement/service/apim-non-prod1/apis/uat-dnaapi-person"
        name                  = "uat-dnaapi-person"
        # (13 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }
  # azurerm_api_management_api.prod_dnaapi-person has been changed
  ~ resource "azurerm_api_management_api" "prod_dnaapi-person" {
        id                    = "/subscriptions/bob/resourceGroups/rg-apim2/providers/Microsoft.ApiManagement/service/apimg-prod2/apis/prod-dnaapi-person"
        name                  = "prod-dnaapi-person"
        # (13 unchanged attributes hidden)
        # (2 unchanged blocks hidden)
    }

[dduleep]

This really helps when PLAN redirects to an approver who doesn't have access to the real resource.
please enable flag for ”Objects have been changes outside of Terraform” and unchanged blocks hidden new features.

[perobertson]

Is there any way to see the full output now that TF_X_CONCISE_DIFF has been removed?
A flag to specify the number of surrounding lines or attributes would also work here.

For context this is with the ordered_cache_behavior on the aws_cloudfront_distribution that is causing the most issues. When adding in a new cache behaviour you have no context to know if you created the order correctly. The only workaround I have come up with is to apply the change, go check if its correct, and then adjust the code before applying again.

[dduleep]

It is almost 10 months completed, No milestone defined for this bug.

[imaginarynik]

This is terrible! I came here looking for answers and it seems the guys at TF are not listening to the requests!!

[jchristner-trustwave]

Modifying my comment from the recently linked and closed issue.

I agree we need a -verbose option. In our situation, a concise diff is problematic when updating an aws_ecs_task_definition json blob for the container_definitions variable, and that blob contains multiple containers (sidecars). It's not possible to tell which sidecar/container definition a change is in without the full diff of the full json blob or a smarter concise diff which can grok a json blob and include the relavent context. While the non-concise diff may be huge, it's still incredibly helpful to validate the correct changes are being applied in all cases (even if they're more obscure).

[imaginarynik]

The only thing I could do was rollback to v0.15.3...because v0.15.4 is here the "feature"/problem about extra unnecessary info began..
According to me, all that we gotta take care of is this: whether my config and reality is matching...I don't care what the terraform records in tfstate file..and then screams when it sees a wrong order in the reality....say order in json, for instance...

[vojkny]

I came here because I couldn't see exactly what is going to happen and only turn to closed #28906. I don't see a reason for hiding things and not allowing people see exact plan.

[ytitov]

Came here for a solution, looking for this feature.

[DoctorPolski]

This issue currently (17th November 2021) has the label: enhancement which further confirms the ineptitude of the maintainers.

The issue being discussed is clearly that changes made by Hashicorp have broken required functionality making the current version of Terraform potentially unusable for resources containing repeatable blocks. There may of course be many other examples.

Can this issue be given the labelling and therefore the priority it needs please?

[dduleep]

We have CI/CD pipelines for too sensitive network changes. Every network changes plan is going through many higher management approval processes as it is critical to the entire business. Somehow trim plan does not help out for understanding the changes

example

code before update

resource "azurerm_virtual_network" "example" {
  name                = "virtualNetwork1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  address_space       = ["10.0.0.0/16"]
  dns_servers         = ["10.0.0.4", "10.0.0.5"]
}

Code after update
Added "10.0.0.4" as new DNS server

resource "azurerm_virtual_network" "example" {
  name                = "virtualNetwork1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  address_space       = ["10.0.0.0/16"]
  dns_servers         = ["10.0.0.4", "10.0.0.5", "10.0.0.4"]
}

Terraform PLAN

resource "azurerm_virtual_network" "example" {
# (3 unchanged blocks hidden)
 ~ dns_servers         = [
# (2 unchanged blocks hidden)
+ 10.0.0.4
]
}

Approver does have a sense of existing DNS to Approve or Reject the CI/CD pipeline just verifying only the above terraform plan.

[dwangfaulkner-fn]

As another example of why this is critical, I just encountered the following plan today for a change to a GCP IAM policy:

                  ~ {
                      ~ members = [
                            "group:<group-name>",
                          - "serviceAccount:<service-account-name>",
                        ]
                        # (1 unchanged element hidden)
                    },

It's very frustrating to not be able to see that particular "unchanged" element, since it's quite important to know which role the service account is being removed from!

[rquadling]

Another example that caused a small panic initially. An AWS account has been (correctly) suspended pending closure within the AWS console. Terraform reported this as follows:

  # aws_organizations_organization.xxxx1_organisation has changed
  ~ resource "aws_organizations_organization" "xxxx1_organisation" {
      ~ accounts                      = [
            # (10 unchanged elements hidden)
            {
                arn    = "arn:aws:organizations::123456789012:account/o-abcdefghij/000000000001"
                email  = "Xxxx.Yyyy+nt-production-compute@xxxx1.co.uk"
                id     = "000000000001"
                name   = "nt-production-compute"
                status = "ACTIVE"
            },
          ~ {
              ~ status = "ACTIVE" -> "SUSPENDED"
                # (4 unchanged elements hidden)
            },
            {
                arn    = "arn:aws:organizations::123456789012:account/o-abcdefghij/123456789012"
                email  = "Xxxx.Yyyy+management@xxxx1.co.uk"
                id     = "123456789012"
                name   = "management"
                status = "ACTIVE"
            },
            # (2 unchanged elements hidden)
        ]
        id                            = "o-abcdefghij"
      ~ non_master_accounts           = [
            # (10 unchanged elements hidden)
            {
                arn    = "arn:aws:organizations::123456789012:account/o-abcdefghij/000000000001"
                email  = "Xxxx.Yyyy+nt-production-compute@xxxx1.co.uk"
                id     = "000000000001"
                name   = "nt-production-compute"
                status = "ACTIVE"
            },
          ~ {
              ~ status = "ACTIVE" -> "SUSPENDED"
                # (4 unchanged elements hidden)
            },
            {
                arn    = "arn:aws:organizations::123456789012:account/o-abcdefghij/000000000002"
                email  = "Xxxx.Yyyy+devops@xxxx1.co.uk"
                id     = "000000000002"
                name   = "devops"
                status = "ACTIVE"
            },
            # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

So all we know here is that an account was suspended. We know at least 2 of the accounts that were NOT suspended, but even with the most optimistic interpretation of this pre-plan output, there's a distinct lack of useful information being presented. And because it is really quite useless, it devalues the output to a point that who knows if it is worth reviewing.

And once you review it and realise something has changed, the amount of time you now need to spend on identifying exactly what has changed ... it all adds up to a bad situation that is easily solved by having some small options around the diff display.

[piersf]

Is there any fix to this? We can't trust our terraform code anymore because of this bug basically.

I'm running Terraform version 1.1.4 and AWS Provider version 3.70.0 and still seeing this issue.

[dduleep]

Hi All,
Do we have any alternative or workaround to get the full Verbose Plan?
terraform state show for remote state or any other options?
It is quite hard to justify as an issue bing open long time. 😒

[deiga]

@alisdair I'm tagging you since you are the last confirmed contributor to have seen this. Could we get anyone from Hashicorp to give a statement on this issue? As shown in multiple examples, the current diff format is creating a lot of problems. We need to be able to trust the tools we use and Terraform is eroding that trust by not being verbose enough.

[crw]

Walking through the various comments with @alisdair, there are three categories of requests in this thread. First, the original enhancement request for a verbose plan output. Second, legitimate bugs in the current plan output code. Third, other enhancement requests.

I want to start by enumerating the bugs identified in comments. If the original commenter or another contributor is facing one of these issues, could please open these as discrete issues following the issue format template? That will make it easier to get each discrete bug resolved and would be greatly appreciated by the team. edit: these have now been fixed by #30685.

Bugs:

• Feature Request: Verbose option for plan #27547 (comment) (@techdragon)
• Feature Request: Verbose option for plan #27547 (comment) (@bomb-on)
• Feature Request: Verbose option for plan #27547 (comment) (@chadcarlton)
• Feature Request: Verbose option for plan #27547 (comment) (@dduleep)
• Feature Request: Verbose option for plan #27547 (comment) (@rquadling)

@alisdair notes that many of those above issues are similar, in that Terraform ought to be rendering fields called id or name, but for some reason is eliding them.

Two comments look like a similar enhancement request: that the provider should be able to set a flag on a resource attribute that marks it as an "identifier," and that identifier should always be printed in the plan output. @alisdair is looking into whether or not there is an existing enhancement request for this, if not we will create one and update this issue.

• Feature Request: Verbose option for plan #27547 (comment) (@alkar)
• Feature Request: Verbose option for plan #27547 (comment) (@dwangfaulkner-fn)

As for the original enhancement request per the original issue, there is currently no commitment to implement a verbose plan option in the near future. It would be very difficult to implement in the current design, and so plan output will likely not significantly change without the current implementation being overhauled. If this changes, we will be sure to update this ticket.

[mattsfrey]

As for the original enhancement request per the original issue, there is currently no commitment to implement a verbose plan option in the near future. It would be very difficult to implement in the current design, and so plan output will likely not significantly change without the current implementation being overhauled.

This is a pretty terrible outcome to be frank. The plan diffs are a critical component that this tool revolves around, to not allow devs to see the entire plan output when desired seems like a major failure in the core design. For debug purposes there will always be use cases where one would wish to see the entire specification for a resource. Very bummed to see this, count me in the ranks of those who would wish to see this overhaul, IMO this direction was not well thought out and greatly hinders the usefulness of the tool.

[alkar]

Two comments look like a similar enhancement request: that the provider should be able to set a flag on a resource attribute that marks it as an "identifier," and that identifier should always be printed in the plan output. @alisdair is looking into whether or not there is an existing enhancement request for this, if not we will create one and update this issue.

This would be a very welcome change that I think should help with most of the issues we see with regards to the concise diff. However, it still relies on the providers to implement this properly. I still think there's value in being able to produce a full plan as an exception, as @mattsfrey mentions above I very often have to state pull in order to look at details that I could previously see through a verbose plan.

[alisdair]

#30753 is tracking the inability for Terraform to display identifying information for resources or substructures in which id and name are not present or not identifying.

#30685 has been merged as a fix for the bug described by five cases reported in this thread, and will be released in Terraform 1.1.8.

[crw]

We very much appreciate the feedback in this thread. For the time being I am going to lock it, primarily to encourage any new issues in the existing system to be reported discreetly. For those who would like to add their voice to this request, please continue to vote up the issue with the 👍 emoji on the issue description. Thanks for all the feedback on this issue, we do truly appreciate it.