Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A way to hide certain expected changes from the "refresh" report ("Objects have changed outside of Terraform") #28803

Closed
petkaantonov opened this issue May 25, 2021 · 91 comments
Labels
cli enhancement v0.15 Issues (primarily bugs) reported against v0.15 releases

Comments

@petkaantonov
Copy link

petkaantonov commented May 25, 2021

After upgrading to 0.15.4 terraform reports changes that are ignored. It is exactly like commented here: #28776 (comment)

Terraform Version

Terraform v0.15.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.42.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Terraform Configuration Files

resource "aws_batch_compute_environment" "batch_compute" {
  lifecycle {
    ignore_changes = [compute_resources[0].desired_vcpus]
  }

...

  compute_resources {
...
  }
}

resource "aws_db_instance" "postgres_db" {
  ...

  lifecycle {
    prevent_destroy = true
    ignore_changes = [latest_restorable_time]
  }
}

Output

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.db.aws_db_instance.postgres_db has been changed
  ~ resource "aws_db_instance" "postgres_db" {
        id                                    = "db"
      ~ latest_restorable_time                = "2021-05-25T10:24:14Z" -> "2021-05-25T10:29:14Z"
        name                                  = "db"
        tags                                  = {
            "Name" = "DatabaseServer"
        }
        # (47 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # module.batch_processor_dot_backend.aws_batch_compute_environment.batch_compute has been changed
  ~ resource "aws_batch_compute_environment" "batch_compute" {
        id                       = "batch-compute"
        tags                     = {}
        # (9 unchanged attributes hidden)

      ~ compute_resources {
          ~ desired_vcpus      = 0 -> 2
            tags               = {}
            # (9 unchanged attributes hidden)
        }
    }

Expected Behavior

No changes should be reported because they are listed in ignored changes.

Actual Behavior

Changes are reported.

Steps to Reproduce

Change any resource outside of terraform and see that terraform apply reports changed even when they should be ignored.

Additional Context

References

@petkaantonov petkaantonov added bug new new issue not yet triaged labels May 25, 2021
@jbardin jbardin changed the title Terraform 0.15.4 ignore changes no longer working ignore_changes does not suppress drift information May 25, 2021
@jbardin jbardin added cli enhancement v0.15 Issues (primarily bugs) reported against v0.15 releases and removed bug new new issue not yet triaged labels May 25, 2021
@apparentlymart
Copy link
Member

Hi @petkaantonov! Thanks for opening this feature request.

The current behavior is reflecting the long-standing behavior that Terraform does still detect and incorporate remote objects into the state, but then while producing a plan it ignores differences between the configuration and the state. The difference reported in your example is a difference between the prior state and the current remote object, and ignore_changes has never affected that situation but that fact was less visible before because Terraform just silently updated the state rather than reporting it.

While it might seem immaterial whether Terraform updates the state or not here, it can result in a change in behavior of your configuration if any other expressions in the module refer to that value. To be specific, if you had any reference to aws_batch_compute_environment.batch_compute.compute_resources[0].desired_vcpus elsewhere in your module then they would return 2 rather than 0 after detecting this change, and so that is what Terraform is reporting here, in case that ends up being useful context for understanding which actions Terraform proposes (or doesn't propose) in the plan.

I assume that in your case this doesn't really matter much, because you don't have any references to aws_batch_compute_environment.batch_compute.compute_resources[0].desired_vcpus elsewhere in your module and so you don't actually care what's reported in the state. One potential improvement we could consider for this mechanism is for Terraform to try to only report changes to resources that have other expressions referring to them, since changes to a value that nothing refers to can't possibly affect the behavior of the configuration.

However, such a rule is easier to say than to implement because what we've done here is just expose in the UI some long-standing Terraform behavior that was previously invisible, and so changing that behavior at this late state will likely require a lot of research to make sure that the changes don't break use-cases we're not currently aware of. The current output is an honest and correct account of Terraform's behavior, and so I think we need to consider here whether the right thing to do is change Terraform's behavior (which, for something this fundamental, would be challenging to do at this late stage in Terraform's life) or to change the UI to fudge the details a little so that it leaves hidden some details that surface inconvenient truths that don't actually affect configuration behavior.

We won't be able to make any significant changes in this regard in the near future, because the scope of this project was just to be more explicit about what Terraform was already doing rather than to change how Terraform behaves, but we'll use this issue to represent the need and consider what we might change in future releases.

Thanks again!

@petkaantonov
Copy link
Author

I see. In that case, would it be possible to add a separate way to ignore these? e.g. ignore_drifts or something like that? Wouldn't that be easy to implement?

@apparentlymart
Copy link
Member

apparentlymart commented May 25, 2021

The refresh operation doesn't currently really consider the resource configuration at all -- it's job is to synchronize the existing state with remote objects -- so making it react to anything new in the resource configuration is not a trivial design decision.

We also know from our experiences with ignore_changes that this idea of partially ignoring changes to parts of objects is generally not as simple as it first appears: there are lots of resource types where two or more attribute values are connected to each other in some way, and so ignoring one without ignoring the other can make the result inconsistent. For current ignore_changes that sort of inconsistency isn't a massive problem aside from being a bit confusing, because during planning we're just proposing some changes and not actually changing anything, but doing that during refresh might well lead to a situation where the object in the state is no longer a valid input to the provider that's managing it, which would likely lead to errors or crashes where a provider has to deal with broken input in a location where it never did before.

@petkaantonov
Copy link
Author

Could the refresh operation be made configurable in some other way? like an .ignorefile There is probably going to be 3rd party solution if it's not added anyway.

Just some way to ignore these is important, otherwise the output is not as useful because it becomes just manually ignored noise and people become conditioned to ignore it regardless of what it says.

@apparentlymart
Copy link
Member

I don't currently have any ready-to-implement designs to address this feedback. We'll need to do some more research and design work in order to address it. Until then, indeed if your system routinely makes changes to objects outside of Terraform as part of its regular function then the feature in its current state will not be so useful for you,

Hopefully it can still answer the sorts of questions it is aiming to solve, though: if Terraform proposes making a change that doesn't seem justified by a corresponding change in configuration, you can refer to the note about detected changes to see if any of the changes it detected are the explanation. If the actions Terraform is proposing match what you were expecting anyway, then the changes detected are just some extra information that you can safely ignore.

@rhgreene
Copy link

rhgreene commented May 26, 2021

In situations where the user is forced to add an ignore - to stop Terraform from chasing it's own tail - ie with the use of an aws_autoscaling_attachment (where it is documented that you must use an ignore in corresponding autoscaling groups https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_attachment), I think this behaviour constitutes a bug. The drift output states that "Objects have changed outside of Terraform" when in reality the changes are made within terraform and have had to be ignored for reasons that might be justified as a bug in their own right.

Choosing to output drift before taking into account ignores thus creates potentially significant noise when using Terrafom within it's documented bounds.

@koalalorenzo
Copy link

I can relate to this issue a lot, our daily work now has a new pain: filtering out the noise of ignored changes.
"muting" the output would help A LOT! :)

anyway, thank you guys for the good work, keep up with it! 👍

@apparentlymart apparentlymart changed the title ignore_changes does not suppress drift information A way to hide certain expected changes from the "refresh" report ("Objects have changed outside of Terraform") May 26, 2021
@bschaeffer
Copy link

bschaeffer commented May 26, 2021

This feels like a pretty invasive change. I second what @koalalorenzo said.

The main problem with this change is that we lean on our large team of engineers to inspect their own changes carefully on a per-team basis and apply those changes without constant infra approval. Even when diffs are small, though, we can sometimes misunderstand the plan which can lead to some unfortunate consequences. Not everyone understands the inner workings of terraform, or even the best practices, so there is always going to be mistakes. This isn't terraform's fault, we all need to learn the tool a little bit better.

Now, though, we have to sift through an entire block of output and tell our team to just ignore it up to a point. Its just going to lead to less clear plan output that people put less effort into reviewing. I think its going to erode trust, too, even though I suspect the opposite is the goal.

For instance, take a google_container_node_pool resource. If its auto-scaling, every single time you apply the project you get a new node_count attribute diff that is wholly un-actionable. If somebody sees that every single time, but I tell them to ignore it because it should absolutely be ignored, whose to say when they get a -/+ change on that resources (maybe by accident or lack of understanding) they aren't going to assume its also ignorable.

TBH, I have ignored many attributes in terraform because we may change them outside of terraform and have come to rely on the fact that it really doesn't and shouldn't care about the changes to attributes you don't specify. IDK. Ranting a bit.

I have to be totally honest I see no point in opting everyone into this for every resource automatically. I don't understand the problem that its solving and unless there is a way for it to be completely filtered out I would love to see this change reverted in the next release.

(I also don't think the solution is ignore_attributes.)

@sofam
Copy link

sofam commented May 27, 2021

I am very surprised about this change and it adds a ton of noise to the already not exactly clear and concise plans. The extra verbosity adds a lot of cognitive load which risks at best to make people glance over the plans and at worst make them ignore them completely.

I really, really don't understand the reasoning here.

Again, other than that, thank you for a great product over the years!

@bschaeffer
Copy link

bschaeffer commented May 27, 2021

I can get a little worked up and forget that a lot of people put in a lot of work to get terraform where it is today. I don't want to belittle that. People like @apparentlymart (who helped us in the forums when we were starting to write our own internal terraform provider plugins) are awesome. Terraform is awesome. Obviously my job is infinitely easier because of it. ❤️

As for this feature, it was in the CHANGELOG and I did glance over it, so its on me that I upgraded to 0.15.4. I just wanted to register my feeedback that I really wish this was separate from the plan, or that it was opt-in when running plans/applies, or that it was even opt in at a module or resource level (a lifecycle { show_drift = true } attribute?).

@geoff-reason
Copy link

Just adding another resource type that now always appears in my terraform output - aws_efs_file_system.

(from #28845 which I will mark as a duplicate). EFS file systems are supposed to grow and change, as many other types of resource they are explicitly designed that way.

With no way to ignore, the plan/apply output is now so noisy I've gone back to terraform 0.15.3 as wading through tens of modified resources on every single plan/apply just to catch the one that might be important is not a workable pattern for me.

@lra
Copy link

lra commented Jun 3, 2021

Since terraform 0.15.4, I get confused messages from my team every single day. I can't imagine the impact this change had globally.

Please add an option to disable displaying those by default and go back to the previous behavior, with maybe an explicit way to get more details about the changes only when needed (e.g a debug mode).

This is really confusing right now, and someone needs to understand the internals of terraform to appreciate the difference between a drift and remote changes.

@joeaawad
Copy link

joeaawad commented Jun 3, 2021

One thing that surprised me a lot about this change is that the Note: Objects have changed outside of Terraform still shows up if you do terraform plan -input=false -output=tfplan >/dev/null && terraform show tfplan. I had hoped that this note would be output as part of the Refreshing state ... messages and as such would be hidden, but it turns out that the note is part of the saved plan file. If this was changed so that the note is output with the Refreshing state ... messages instead of being saved inside of the plan, that would make a big difference for the ability of users to silence that note and anyone who was already silencing the refreshing state messages would automatically get this new note silenced as well.

As near as I can tell, the only way to currently silence the warning is to run terraform refresh >/dev/null && terraform plan -input=false -refresh=false, which has several downsides that @jbardin and @apparentlymart pointed out in #27214.

@hashicorp hashicorp deleted a comment from roslee77 Jun 3, 2021
@hashicorp hashicorp deleted a comment from roslee77 Jun 3, 2021
@chloeruka
Copy link

chloeruka commented Jun 4, 2021

This is really confusing right now, and someone needs to understand the internals of terraform to appreciate the difference between a drift and remote changes.

Agreed, this bit us earlier. Another example is that AWS autoscaling groups will always have "external" changes to attributes such as desired_count from normal scaling behaviour so the refresh-only state plan will show drift. I was going to report this to the AWS provider, but it sounds like they might not yet have a way to fix this. If there are underlying architecture changes that need to be carefully considered, I would certainly prefer to suppress these drift notifications until they can be made more useful.

As the below is an ignored attribute, the resource is not affected by Terraform and the message amounts to noise:
Example:

  # aws_autoscaling_group.example-scaling-group has been changed
  ~ resource "aws_autoscaling_group" "example-scaling-group" {
      ~ desired_capacity          = 50 -> 36
        id                        = "example-scaling-group"
        name                      = "example-scaling-group"
        # (20 unchanged attributes hidden)

A person reviewing this plan has to already know the attribute is ignored on that resource or read "below the line" in order to confirm the resource is not affected. Another way of looking at this is as a formatting issue, since the change notification looks very similar to the actual plan, and it took me and the rest of my team a while to realise it's a separate section.

@ryanking
Copy link
Contributor

ryanking commented Jun 7, 2021

Another example - okta_app_oauth supports users which can also be configured by a separate resource. If you use the separate resource or some out-of-band way to manage the users, you will always see a change here.

@drdamour
Copy link

the size attribute of azuredevops_git_repository is another of these dynamic fields that i really don't need to see every time i plan/apply. was disappointed ignore didn't just silent it, makes sense cause someone somewhere has some TF that does something based on size changing i'm sure...thinking something like a new lifecycle prop like

lifecycle {
  use_current = [
    size
  ]
}

that tells TF to always update state from current value without reporting it as a difference.
could also see it being driven by providers, since there are cases where someone may specify a size for an azdo repo, so azdo provider could report size as this new class of property that doesn't need to indicate change when it's say set to null but should if it's set to a non null value (course the latter case is redundant since you'll see that in the plan)

@yongzhang
Copy link

same issue here:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  ~ resource "aws_ecr_repository_policy" "repository" {
        id          = "dockerhub/nginx"
      ~ policy      = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws-cn:iam::123:root",
                                "arn:aws-cn:iam::789:root",
                              - "arn:aws-cn:iam::456:root",
                                "arn:aws-cn:iam::7890:root",
                                "arn:aws-cn:iam::78901:root",
                              + "arn:aws-cn:iam::123:root",
                              + "arn:aws-cn:iam::456:root",
                            ]
                        }
                        # (3 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        # (2 unchanged attributes hidden)
    }

@koalalorenzo
Copy link

koalalorenzo commented Sep 17, 2021

It is such a big frustration that the other day I rejected a valid PR with atlantis badly because I got confused what were the changes. Please make this output OPTIONAL!

Who do we contact to make this disappear in our logs? Is this ever going to be fixed? What can we do to make this a priority? Is there any contributor or Hashicorp employee taking care of UX changes in Terraform that we can ask to check this issue?

Here is a cute doggo asking for help:

[kmoe: edited to remove large image]

@serjaum
Copy link

serjaum commented Sep 17, 2021

same here! It is taking a lot of time to explain to the engineers that this output does not reflect any changes in our infrastructure.

And I'm afraid if the people start to ignore the plan because they are thinking that all changes are only in the state file

@jbscare
Copy link

jbscare commented Sep 17, 2021

Yet another example:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

  # aws_elb.snip has been changed
  ~ resource "aws_elb" "snip" {
        id                          = "snip"
      ~ instances                   = [
          - "i-0245934821285edc7",
          - "i-054d85efadb1d4c0a",
          - "i-0cdda3b2595fe3281",
            # (3 unchanged elements hidden)
        ]
        name                        = "snip"
        tags                        = {
            "Name"                = "snip"
        }
        # (13 unchanged attributes hidden)


        # (2 unchanged blocks hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

The following plan will 100% not ever include actions to undo or respond to those changes, because we've explicitly and deliberately in our configuration told Terraform not to manage the instances in the ELB. (Sometimes other things need to take instances in and out, e.g. Ansible while deploying things.)

It says right there in our configs that Terraform should 100% not ever care what instances are in the ELB. Our configuration is being ignored, and Terraform is now caring (albeit only at the level of a warning) about things that we explicitly and deliberately told it not to care about.

This is not detecting drift.

This is a bug.

@sergei-ivanov
Copy link

I have a suggestion. How about Terraform introduces a new sub-command:

terraform state diff

...and the output of that command will be the same as we are seeing in "Terraform detected the following changes made outside of Terraform" section since Terraform 1.0.

Then we can completely remove that state diff from terraform plan output. Because all we normally care about are the plan changes that will result in create/update/destroy actions on the resources. If the state difference has no impact on the plan, it is almost always noise. If there are detected changes in the plan, Terraform may suggest running terraform state diff separately, which may give us some additional diagnostic information if and when needed.

@ChristophShyper
Copy link

That's not a great solution, as it only solves few issues.
Sometimes your infrastructure can be modified externally, by a Lamba or some manual action. With your solution you completely lose that information.
This thread wants to introduce a way of hiding some specifically defined properties. Which always show up, but are not caused by external factors, rather by how some providers work.

@sergei-ivanov
Copy link

Terraform will still check for drift between the state and the reality. It has always been doing a refresh before plan. And I believe it's always been the case that any changes outside of Terraform could trigger plan changes (i.e. resource modification). BUT when I am looking at the plan, I am mostly interested in what changes Terraform is going to make. It may choose to undo some changes made externally, or pick up some external changes (via computed resource attributes or data sources). What I am not really interested in is the state drift that might cause plan changes, but in reality it does not. The state diff output as part of the plan would be more useful/helpful if it only included the differences that resulted in plan changes and ignored all the others.

If someone wanted to check the difference between the last applied state and the reality (i.e. any external changes since last terraform apply or terraform refresh), a new terraform state diff command would be perfect for that. Or add a flag -display-state-diff=true|false to terraform plan and terraform apply.

@justinTM
Copy link

justinTM commented Oct 1, 2021

At this point my jobs are failing and I can't even see why, because across tens of templated JSON files totaling a few hundred thousand LOC, the (ridiculously generous) maximum log file size in GitLab of 4MB is being exceeded with (and I will be flippant here) COMPLETELY worthless Terraform information.

It's been all year and there is still not even a single flag to suppress expected changes? I know HashiCorp went through an employee-quitting phase but this is like an intern task and is directly impacting production. Should've stayed on 0.15 but there is no going back at this point.

@kvz
Copy link
Contributor

kvz commented Oct 13, 2021

I think the current behavior is actually a bit dangerous, because when faced with walls of drift, which is now the new normal for any somewhat large install, for each proposed change, the brain trains to ignore this noise and hit yes. It will become ever easier to miss unintended changes as you use this more.

@leonsmith
Copy link

@kvz makes a real point. Never-mind try to explain to developers who interact with terraform infrequently "ohh ignore this bit but don't ignore this bit" it's asking for accidents to be happen.

@alexrosenfeld10
Copy link

Since terraform 0.15.4, I get confused messages from my team every single day. I can't imagine the impact this change had globally.

Please add an option to disable displaying those by default and go back to the previous behavior, with maybe an explicit way to get more details about the changes only when needed (e.g a debug mode).

This is really confusing right now, and someone needs to understand the internals of terraform to appreciate the difference between a drift and remote changes.

I feel very much the same way I have to scroll through hundreds of useless etag diffs just to find my one resource I actually changed. This 100% needs fixing or some way of hiding.

@charles-dexter-ward
Copy link

Same problem, updating from 0.13.5 to 1.0.9.

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.lambda_exec_role.aws_iam_role.iam_role has been changed
  ~ resource "aws_iam_role" "iam_role" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ Service = [
                              - "s3.amazonaws.com",
                              - "ssm.amazonaws.com",
                                "lambda.amazonaws.com",
                              + "ssm.amazonaws.com",
                                "edgelambda.amazonaws.com",
                              + "s3.amazonaws.com",
                            ]
                        }
                        # (2 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        id                    = "lambda_exec_role"
        name                  = "lambda_exec_role"
        tags                  = {
            "env" = "test"
        }
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_lambda_layer_version.pkgupdates has been changed
  ~ resource "aws_lambda_layer_version" "pkgupdates" {
      + compatible_architectures = []
        id                       = "arn:aws:lambda:eu-west-1:497776581864:layer:pkgupdates:23"
        # (9 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

@imaginarynik
Copy link

@apparentlymart @jbardin Is there an update on this issue, please?!

@lindhe
Copy link

lindhe commented Oct 22, 2021

I just spent a good hour debugging why my diff was so strange. Turns out it wasn't, it was just Terraform playing mind games with me. Not funny.

@imaginarynik
Copy link

imaginarynik commented Oct 22, 2021

Worst part...it's not even consistent..yesterday I had completely different "change detections" everytime i ran terraform plan...it was as if, the terraform missed results fom the API, and didn't report it..then the next time it detected it..and the next time it didn't..and so on...this was on v0.15.5..i rolled back to v0.15.3..terrible implementation I must say...it's the classic feature vs bug...as rightly pointed out by the others..this is something one would want to see in DEBUG logging, not in default output...Terraform team has opted a completely wrong track, in my opinion...I mean as developer, what do we care about? Just our config and the infra/reality...1:1 mapping there...who cares what Terraform wants to store in its state file...if it sees a difference between what it stores and what sees in reality, it's Terraform's problem, not mine...I just need my code to be in 1:1 sync with reality...the intermediate statefile is Terraform's headache...

@favoretti
Copy link

FWIW, I think functionality is handy, but as others I'd rather have it as a separate command rather than standard plan part.

@kvz
Copy link
Contributor

kvz commented Oct 25, 2021

Or under a prompt:

75 resources changed since last invocation, do you want to inspect this drift?

with a flag to always pass yes or no via --drift and --no-drift, or similar.

@dulakm
Copy link

dulakm commented Oct 25, 2021

This issue got so many comments since May as other open Terraform issues since 2015.

Based on this it must be obvious how pressing it is and how many people are experiencing problems in their day-to-day work due to this. There are tons of great suggestions and proposals in this thread, could someone please take a look at it already? 🙂

@JordanP
Copy link

JordanP commented Oct 25, 2021

90% of comments are like "yeah me too" or "that sucks". Where the PRs guys ? Your company depends on Terraform, your a professional ? Send some PRs then. It's open source, not everybody can leech.

@kchristensen
Copy link

90% of comments are like "yeah me too" or "that sucks". Where the PRs guys ? Your company depends on Terraform, your a professional ? Send some PRs then. It's open source, not everybody can leech.

6562466

**Note:** Due to current low staffing on the Terraform Core team at HashiCorp, **we are not routinely reviewing and merging community-submitted pull requests**. We do hope to begin processing them again soon once we're back up to full staffing again, but for the moment we need to ask for patience. Thanks!

@loozhengyuan
Copy link

@JordanP I think we are really here to convince the Terraform Core team that the community needs a better solution. There is not much point if we submit a PR if the team is not keen to merge it.

#28803 (comment)

We won't be able to make any significant changes in this regard in the near future, because the scope of this project was just to be more explicit about what Terraform was already doing rather than to change how Terraform behaves, but we'll use this issue to represent the need and consider what we might change in future releases.

@kvz
Copy link
Contributor

kvz commented Oct 25, 2021

90% of comments are like "yeah me too" or "that sucks". Where the PRs guys ? Your company depends on Terraform, your a professional ? Send some PRs then. It's open source, not everybody can leech.

I think hashicorp indicated they don't have the bandwidth to review community PRs right now. And if they had, we should ask how they want to address this, if at all. The code itself to allow to skip showing this, will be a oneliner so that is not what is blocking this

@apparentlymart
Copy link
Member

Hi all! Thanks for all the feedback here.

It's clear that there are lots of opportunities to improve the signal to noise ratio of the current change detection. As I mentioned right back at the top of this discussion, this is Terraform making explicit some behaviors that were previously implicit and thus a common cause for confusion. However, it's also clear that various particular provider features, along with the historical confusing design of ignore_changes, have led to the new behavior being confusing in quite a different way.

Two specific things that the Terraform team is researching to improve this feature are:

  • A new feature that does what everyone thinks ignore_changes does: declare that a particular resource attribute is not relevant to the current configuration and thus not interesting to track.

    What ignore_changes actually does is tell Terraform to ignore changes to the configuration, which unfortunately means that it explicitly doesn't ignore changes in the remote system (which is the source of record). We can't change the behavior of ignore_changes due to the v1.0 compatibility promises, but we could introduce a new feature that makes a stronger statement, which would include making Terraform consider changes to it to be totally immaterial, and thus not mention them.

  • Heuristics to detect better whether any of the proposed changes (the result of the plan) seem likely to have been caused by one of the detected changes (the result of refreshing), and thus automatically effectively infer by static analysis which changes are relevant to report.

    This would effectively be an automatic version of the explicit feature I described in the previous point, avoiding the need to explicitly annotate everything and instead saving explicit annotations only for situations that are too complicated for Terraform's static analysis to understand automatically.

Both of these features are cross-cutting and therefore not something we can just casually implement without careful design first. However, we are indeed actively investigating both and hope to have improvements to share in a forthcoming release.

Since this discussion has got quite heated and it seems like we've already gathered sufficient feedback, I'm going to lock this discussion for now in order to reduce the noise for the many folks who are following this issue. We'll unlock it again when either we have more news to share or if we need some specific feedback in order to shape solutions like what I described above.

@hashicorp hashicorp locked as too heated and limited conversation to collaborators Oct 25, 2021
@jbardin
Copy link
Member

jbardin commented Mar 18, 2022

Hello All!

Here to deliver the latest update! With the addition of #30486, we hope to address the concerns raised here, while also taking into account the users who do desire the refresh report in order to help understand the changes within a plan and detect unexpected changes.

Starting with v1.2, the goal for the refresh report is that only external changes which may have contributed to changes in the plan will be shown. This means in most cases, unused attributes changing outside of terraform will not show up in the normal plan output. If there are no changes in the plan, no external changes will be shown in the CLI at all. All refresh information is still stored within the plan, and if a user wants to see all external changes of resources in the CLI, a refresh-only plan can be used.

For more details on the change, see #30486. For any questions or discussion, feel free to use the community forum.

Thanks!

@jbardin jbardin closed this as completed Mar 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cli enhancement v0.15 Issues (primarily bugs) reported against v0.15 releases
Projects
None yet
Development

No branches or pull requests