Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Plugin SDK to Plugin Framework migration tests for meta data sources #27378

Closed
ewbankkit opened this issue Oct 21, 2022 · 4 comments · Fixed by #27553
Closed

Add Plugin SDK to Plugin Framework migration tests for meta data sources #27378

ewbankkit opened this issue Oct 21, 2022 · 4 comments · Fixed by #27553
Labels
tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.

Comments

@ewbankkit
Copy link
Contributor

ewbankkit commented Oct 21, 2022

Add Plugin SDK to Plugin Framework migration acceptance tests for the meta data sources.

Relates

In particular test for "zero" values of "complex" attributes (lists, sets, maps, nested blocks).

@ewbankkit ewbankkit changed the title Add Plugin SDK to Plugin Framework migration tests for data sources Add Plugin SDK to Plugin Framework migration tests for meta data sources Oct 21, 2022
@github-actions
Copy link

Community Note

Voting for Prioritization

  • Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

  • If you are interested in working on this issue, please leave a comment.
  • If this would be your first contribution, please review the contribution guide.

@ewbankkit ewbankkit added service/meta tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Oct 21, 2022
@ewbankkit
Copy link
Contributor Author

Some thoughts:

@ewbankkit
Copy link
Contributor Author

From internal Slack thread:

Kit Ewbank Oct 24th at 2:17 PM

Good afternoon TF DevEx 👋
As was discussed at last week's Ecosystem/DevEx Office Hours, last week's v4.36.0 AWS Provider release included a number of SDK-to-Framework migrated data sources.
Unfortunately we had a regression (#27372) caused by a null vs. empty ({}) map.
I transcribed this code (v4.35.0)

	if tags != nil {
		if err := d.Set("tags", tags.IgnoreAWS().IgnoreConfig(ignoreTagsConfig).Map()); err != nil {
			return fmt.Errorf("error setting tags: %w", err)
		}
	} else {
		d.Set("tags", nil)
	}

to (v4.36.0)

	if tags != nil {
		data.Tags = flex.FlattenFrameworkStringValueMap(ctx, tags.IgnoreAWS().IgnoreConfig(ignoreTagsConfig).Map())
	} else {
		data.Tags = types.Map{ElemType: types.StringType, Null: true}
	}

The issue was that d.Set("tags", nil) doesn't set tags to null in state, but sets it to {} . That's something to remember for the future, but in trying to ensure that acceptance tests would have caught this I noticed:
We already had a resource.TestCheckResourceAttr(dataSourceName, "tags.%", "0"), check that didn't catch the {} to null regression - Is this because of the whole flatmap/shim mechanism of reading state losing the distinction between null and empty maps?
As part of the hotfix for the regression I added a migration test following the devdot guidelines: https://developer.hashicorp.com/terraform/plugin/framework/migrating/testing#testing-migration (in #27377) but this new test failed to detect any terraform plan difference - Is this is because of the flatmap/shim state behavior or something more general with data sources in that the DS is refreshed before the plan check and so will never show a diff, no matter how big the actual state difference: https://github.com/hashicorp/terraform-plugin-sdk/blob/a096f3a8138e0256b03b133db888034d18170236/helper/resource/testing_new_config.go#L24-L31
Thanks

Brian Flad 1 day ago

We already had a resource.TestCheckResourceAttr(dataSourceName, "tags.%", "0"), check that didn’t catch the {} to null regression - Is this because of the whole flatmap/shim mechanism of reading state losing the distinction between null and empty maps?

I will look into this right now; wouldn’t surprise me if the shims treated the null attribute the same as an empty map :eyeroll:
Unfortunately changing those TestCheck functions to error when the attribute is null (similar to how we started raising errors if you forgot the length sigils) is a much more unfriendly breaking change, but we could still consider it a unfortunate bug fix
I have a feeling a lot of places will probably be checking things like that rather than resource.TestCheckNoResourceAttr() but, I could be mistaken.
:sigh_qoobee:

func testCheckResourceAttr(is *terraform.InstanceState, name string, key string, value string) error {
	v, ok := is.Attributes[key]

	if !ok {
		// Empty containers may be elided from the state.
		// If the intent here is to check for an empty container, allow the key to
		// also be non-existent.
		if value == "0" && (strings.HasSuffix(key, ".#") || strings.HasSuffix(key, ".%")) {
			return nil
		}

hashicorp/terraform-plugin-sdk@57c8f96 was the origin for that, which alludes to my suspicion there

Missing containers were often erroneously kept in the state, but since
the addition of the new provider shims, they can often be correctly
eliminated. There are however many tests that check for a "0" count in
the flatmap state when there shouldn't be a key at all. This addition
looks for a container count key and "0" pair, and allows for the key to
be missing.

There may be some tests negatively effected by this which were
legitimately checking for empty containers, but those were also not
reliably detected, and there should be much fewer tests involved.

So, we essentially get to decide whether to frustrate the ecosystem by making a breaking change or make it opt-in and difficult to discover when it might not do what you’d expect. I’m not sure we want to get into the business of “well if the provider is installed via ProtoV5ProviderFactories/ProtoV6ProviderFactories (probably framework or go based, but not guaranteed) do a different behavior” but that’s another option if we squint really hard. :shifty_eyed_dog:
I guess in order of preference I’d say:

  • Somehow remove that conditional for ProtoV5ProviderFactories/ProtoV6ProviderFactories… which I have no idea how we could detect it correctly in there 😞
  • Provide a new TestCheckResourceAttr() without that 😞
  • Provide an acceptance testing environment variable to control that behavior 😞
  • Remove the conditional entirely, release as a bug fix in a minor version, deal with fallout 😞
  • Remove the conditional entirely, release as a bug fix in a major version (Go makes this a big pain as you have to update version paths everywhere and there’s a ton of other stuff that would ideally go into a major version release) 😞
    I’m not enamored with any of these options

Relating to the second option there about a new TestCheckFunc, we could introduce something quite specific, e.g.
func TestCheckResourceAttrCollectionSize(address string, path string, size int) TestCheckFunc
Which would correctly error if the attribute doesn’t exist and would remove the special .#/.% sigils but that’s quite opt-in and people could get frustrated trying to use that with sdk providers, due to the reason the conditional was added in the first place

I went through the existing sdk issues and didn’t see anything relating to this, so I guess a good first step is creating one if we want to do anything here. I could probably create and test that TestCheckFunc fairly quickly since I’m familiar with how the other ones are unit tested.

I’m quite hesitant to suggest changing the existing TestCheckResourceAttr behavior because these code searches yield lots of potential issues
.# and 0 (7.6k files?): https://cs.github.com/?scopeName=All+repos&scope=&q=language%3Ago+%2FTestCheckResourceAttr%5C%28%5B%5E%2C%5D%2B%2C+%22.*%5C.%23%22%2C+%220%22%5C%29%2F
.% and 0 (4.7k files?): https://cs.github.com/?scopeName=All+repos&scope=&q=language%3Ago+%2FTestCheckResourceAttr%5C%28%5B%5E%2C%5D%2B%2C+%22.*%5C.%25%22%2C+%220%22%5C%29%2F (edited)

Created hashicorp/terraform-plugin-sdk#1085

but this new test failed to detect any terraform plan difference

This is to be expected. Data sources don’t really have a plan per-se, only a refreshed state. Terraform used to (maybe it still does in some cases) compare a prior data source state versus current data source state, but data source schemas are not versioned so that comparison/plan is now removed in most cases to prevent errors that providers cannot avoid.
To test this particular case you’d need to extract the state value in the sdk TestStep and the framework TestStep, then ensure those values are equal
… which is really non-trivial in this case because its all flatmap state

func TestAccDefaultTagsDataSource_EmptyMapCheck(t *testing.T) {
	t.Parallel()

	var tags1, tags2 map[string]string

	Test(t, TestCase{
		Steps: []TestStep{
			{
				ExternalProviders: map[string]ExternalProvider{
					"aws": {
						Source:            "registry.terraform.io/hashicorp/aws",
						VersionConstraint: "4.35.0",
					},
				},
				Config: `data "aws_default_tags" "test" {}`,
				Check: ComposeAggregateTestCheckFunc(
					testExtractResourceAttrMap("data.aws_default_tags.test", "tags", &tags1),
				),
			},
			{
				ExternalProviders: map[string]ExternalProvider{
					"aws": {
						Source:            "registry.terraform.io/hashicorp/aws",
						VersionConstraint: "4.36.0",
					},
				},
				Config: `data "aws_default_tags" "test" {}`,
				Check: ComposeAggregateTestCheckFunc(
					testExtractResourceAttrMap("data.aws_default_tags.test", "tags", &tags2),
					func(s *terraform.State) error {
						if diff := cmp.Diff(tags1, tags2); diff != "" {
							return fmt.Errorf("tags attribute value difference: %s", diff)
						}

						return nil
					},
				),
			},
		},
	})
}

func testExtractResourceAttrMap(resourceName string, attributeName string, attributeValue *map[string]string) TestCheckFunc {
	return func(s *terraform.State) error {
		rs, ok := s.RootModule().Resources[resourceName]

		if !ok {
			return fmt.Errorf("resource name %s not found in state", resourceName)
		}

		if _, ok := rs.Primary.Attributes[attributeName]; ok {
			return fmt.Errorf("attribute name %s found, but is not a map attribute in state", attributeName)
		}

		// Everything in a map will be in the flatmap with the attribute name,
		// a period, then either the size sigil or a string key.
		attributeValuePathPrefix := attributeName + "."
		attributePathSize := attributeValuePathPrefix + "%"

		sizeStr, ok := rs.Primary.Attributes[attributePathSize]

		if !ok {
			// attribute not found
			*attributeValue = nil

			return nil
		}

		size, err := strconv.Atoi(sizeStr)

		if err != nil {
			return fmt.Errorf("unable to convert map size %s to integer: %s", sizeStr, err)
		}

		m := make(map[string]string, size)

		for attributePath, attributePathValue := range rs.Primary.Attributes {
			// Ignore
			if !strings.HasPrefix(attributePath, attributeValuePathPrefix) {
				continue
			}

			if attributePath == attributePathSize {
				continue
			}

			key := strings.TrimPrefix(attributePath, attributeValuePathPrefix)

			m[key] = attributePathValue
		}

		*attributeValue = m

		return nil
	}
}
AWS_REGION=us-east-1 TF_ACC=1 go test -count=1 -run='TestAccDefaultTagsDataSource_EmptyMapCheck' -v ./helper/resource
=== RUN   TestAccDefaultTagsDataSource_EmptyMapCheck
=== PAUSE TestAccDefaultTagsDataSource_EmptyMapCheck
=== CONT  TestAccDefaultTagsDataSource_EmptyMapCheck
    teststep_providers_test.go:1990: Step 2/2 error: Check failed: 1 error occurred:
                * Check 2/2 error: tags attribute value difference:   map[string]string(
        -       {},
        +       nil,
          )
        
        
--- FAIL: TestAccDefaultTagsDataSource_EmptyMapCheck (24.96s)
FAIL
FAIL    github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource    25.212s
FAIL

sorry that’s written in from the sdk helper/resource package, but that’s the best thing I can figure there
(updating the second step to 4.36.1 passes 😛 )
Another cursed way (since we know how its going to fail) to check this is:

func TestAccDefaultTagsDataSource_EmptyMapCheck(t *testing.T) {
	t.Parallel()

	Test(t, TestCase{
		Steps: []TestStep{
			{
				ExternalProviders: map[string]ExternalProvider{
					"aws": {
						Source:            "registry.terraform.io/hashicorp/aws",
						VersionConstraint: "4.35.0",
					},
				},
				Config: `data "aws_default_tags" "test" {}`,
				Check: ComposeAggregateTestCheckFunc(
					TestCheckResourceAttrSet("data.aws_default_tags.test", "tags.%"),
				),
			},
			{
				ExternalProviders: map[string]ExternalProvider{
					"aws": {
						Source:            "registry.terraform.io/hashicorp/aws",
						VersionConstraint: "4.36.0",
					},
				},
				Config: `data "aws_default_tags" "test" {}`,
				Check: ComposeAggregateTestCheckFunc(
					TestCheckResourceAttrSet("data.aws_default_tags.test", "tags.%"),
				),
			},
		},
	})
}

😄

=== CONT  TestAccDefaultTagsDataSource_EmptyMapCheck
    teststep_providers_test.go:1990: Step 2/2 error: Check failed: 1 error occurred:
                * Check 1/1 error: data.aws_default_tags.test: Attribute 'tags.%' expected to be set
        
--- FAIL: TestAccDefaultTagsDataSource_EmptyMapCheck (15.52s)

@github-actions
Copy link

github-actions bot commented Dec 1, 2022

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant