Upgrade TF azurerm v2 and NatGateway pubIP migration #192

dkistner · 2020-11-16T14:52:46Z

How to categorize this PR?

/area open-source
/kind technical-debt
/priority normal
/platform azure

What this PR does / why we need it:
This PR contains migration logic to switch the azurerm Terraform provider version from v1.x.x to version v2.x.x (major version change).

In detail the following changes in the Terraform are required:

Add features field to the Terraform manifest
Remove the route table and network security group references from the azurerm_subnet resource. There are meanwhile associations resources in place, but we need to keep the resource references before we can migrate. Before this migration can be applied all Shoots Infrastructure need to be reconciled by a Gardener Azure extension which contains this PR: Prepare upgrade to Terraform azurerm v2.x.x #161 (shipped with v1.14.0)
The azurerm_subnet resource field address_prefix has been moved to address_prefixes
Added the migration steps for the Gardener managed NAT IP proposed by @ialidzhikov. The TODO(natipmigration) changes can be removed in future release when all Shoot Infrastructure are reconciled with this change. These changes are required to unblock also NatGateway integration – step 2 #54

This PR can only be merged when a new Terraformer with this PR gardener/terraformer#54 is in.

Special notes for your reviewer:
Create an Infrastructure with NatGateway (with an Azure extension not including this PR and an azurerm tf provider of version v1.x.x). Then reconcile the existing Infrastructure an Azure extension which contains these changes.

This Terraformer image can be used for testing: dominickistner/terraformer:azurerm-2360

Release note:

The Gardener Azure provider extension is now using the `azurerm` Terraform provider in version `v2.x.x`

Operators need to ensure that all Azure Shoot `Infrastructure` resource are reconciled with the Gardener Azure provider extension `v1.14.0` (contains a PR which is a prerequisite for this Terraform migration https://github.com/gardener/gardener-extension-provider-azure/pull/161) before applying this version. Please have a special look on hibernated clusters as their `Infrastructure` is might not reconciled for a while.

Terraformer uses now the azurerm provider in version v2.36.0

/invite @ialidzhikov
/invite @kon-angelo

dkistner · 2020-11-16T14:54:49Z

@ialidzhikov Could you have a look? I will remove the draft of this pr gardener/terraformer#54 when we have reviewed this one here.

dkistner · 2020-11-16T15:13:08Z

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

CI/Concourse seems to pull to much from dockerhub

ialidzhikov

I played with it locally and the migration worked well. Looks good generally, thank you! Minor nits inline.

Before this migration can be applied all Shoots Infrastructure need to be reconciled by a Gardener Azure extension which contains this PR: #161 (shipped with v1.14.0)

There is one corner case that Infrastructures are no longer reconciled for hibernated Shoot - see gardener/gardener#2258. So technically there might be an Infrastructure of hibernated Shoot that is not reconciled by provider-azure@v1.14.0.

pkg/internal/infrastructure/terraform.go

ialidzhikov · 2020-11-17T14:27:02Z

We can also run the infrastructure integration test when there is a release of terraformer. Or alternatively you can also update chart/images.yaml with terraformer v1.5.0-dev-b85b19c9d8ce07882aa1b382e26e4560e04b67cc (I am not sure whether the component descriptor will like this)?

dkistner · 2020-11-18T12:46:10Z

I run the infra integration tests locally with my custom terraformer dominickistner/terraformer:azurerm-2360 before I opened the pr. Those succeed.
Btw: Addressed also the requested changes. PTAL.

ialidzhikov · 2020-11-19T08:00:32Z

/lgtm

Before this migration can be applied all Shoots Infrastructure need to be reconciled by a Gardener Azure extension which contains this PR: #161 (shipped with v1.14.0)

There is one corner case that Infrastructures are no longer reconciled for hibernated Shoot - see gardener/gardener#2258. So technically there might be an Infrastructure of hibernated Shoot that is not reconciled by provider-azure@v1.14.0.

@dkistner , what is our last agreement about this? Manual detection of such clusters and manual trigger for reconcile? If that is the case, will you take care about this?

kon-angelo · 2020-11-23T11:34:01Z

docker's rate limit strikes again

ialidzhikov · 2020-11-23T12:02:59Z

docker's rate limit strikes again

Let's switch the base image to eu.gcr.io/gardener-project/3rd/golang:1.15.3. With the g/g@v1.13 vendoring, we will have gardener/gardener#3189 in place.

ialidzhikov · 2020-11-23T12:03:53Z

/test-single

testmachinery · 2020-11-23T12:03:56Z

Testrun: e2e-5j4c2
Workflow: e2e-5j4c2-wf
Phase: Succeeded

+---------------------+---------------------+-----------+----------+
|        NAME         |        STEP         |   PHASE   | DURATION |
+---------------------+---------------------+-----------+----------+
| infrastructure-test | infrastructure-test | Succeeded | 17m45s   |
+---------------------+---------------------+-----------+----------+

ialidzhikov · 2020-11-23T14:19:24Z

/needs rebase

ialidzhikov

/lgtm

kon-angelo

/lgtm

gardener-robot added kind/api-change API change with impact on API users needs/second-opinion Needs second review by someone else labels Nov 16, 2020

gardener-robot requested review from ialidzhikov and kon-angelo November 16, 2020 14:52

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 16, 2020

gardener-robot-ci-2 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 16, 2020

ialidzhikov added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 17, 2020

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 17, 2020

ialidzhikov reviewed Nov 17, 2020

View reviewed changes

pkg/internal/infrastructure/terraform.go Outdated Show resolved Hide resolved

pkg/internal/infrastructure/terraform.go Outdated Show resolved Hide resolved

pkg/internal/infrastructure/terraform.go Outdated Show resolved Hide resolved

gardener-robot removed the needs/review Needs review label Nov 17, 2020

dkistner force-pushed the tf-2-migration branch from 95e4410 to 473efc3 Compare November 18, 2020 12:43

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 18, 2020

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 18, 2020

dkistner mentioned this pull request Nov 18, 2020

Update azurerm provider to v2.36.0 (major version upgrade) gardener/terraformer#54

Merged

gardener-robot added reviewed/lgtm Has approval for merging and removed needs/second-opinion Needs second review by someone else labels Nov 19, 2020

ialidzhikov mentioned this pull request Nov 19, 2020

[ci:component:github.com/gardener/terraformer:v1.4.0->v1.5.0] #195

Closed

dkistner force-pushed the tf-2-migration branch from 473efc3 to 10e12cc Compare November 19, 2020 08:52

dkistner dismissed kon-angelo’s stale review via 32e50a8 November 23, 2020 11:25

dkistner force-pushed the tf-2-migration branch from 10e12cc to 32e50a8 Compare November 23, 2020 11:25

gardener-robot added needs/second-opinion Needs second review by someone else and removed reviewed/lgtm Has approval for merging labels Nov 23, 2020

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 23, 2020

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 23, 2020

kon-angelo previously approved these changes Nov 23, 2020

View reviewed changes

dkistner marked this pull request as ready for review November 23, 2020 14:08

dkistner requested review from a team as code owners November 23, 2020 14:08

gardener-robot added the needs/rebase Needs git rebase label Nov 23, 2020

Upgrade TF azurerm v2 and NatGateway pubIP migration

5298abf

dkistner dismissed kon-angelo’s stale review via 5298abf November 23, 2020 14:29

dkistner force-pushed the tf-2-migration branch from 32e50a8 to 5298abf Compare November 23, 2020 14:29

gardener-robot removed the needs/rebase Needs git rebase label Nov 23, 2020

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 23, 2020

ialidzhikov approved these changes Nov 23, 2020

View reviewed changes

gardener-robot added reviewed/lgtm Has approval for merging and removed needs/second-opinion Needs second review by someone else labels Nov 23, 2020

rfranzke requested a review from kon-angelo November 23, 2020 15:49

kon-angelo approved these changes Nov 23, 2020

View reviewed changes

ialidzhikov merged commit 113bc09 into gardener:master Nov 23, 2020

ialidzhikov mentioned this pull request Nov 25, 2020

Replace deprecated terraform resources #184

Closed

dkistner mentioned this pull request Nov 30, 2021

Cleanup natipmigration logic #406

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade TF azurerm v2 and NatGateway pubIP migration #192

Upgrade TF azurerm v2 and NatGateway pubIP migration #192

dkistner commented Nov 16, 2020 •

edited

dkistner commented Nov 16, 2020

dkistner commented Nov 16, 2020 •

edited

ialidzhikov left a comment

ialidzhikov commented Nov 17, 2020

dkistner commented Nov 18, 2020

ialidzhikov commented Nov 19, 2020

kon-angelo commented Nov 23, 2020

ialidzhikov commented Nov 23, 2020

ialidzhikov commented Nov 23, 2020

testmachinery bot commented Nov 23, 2020 •

edited

ialidzhikov commented Nov 23, 2020

ialidzhikov left a comment

kon-angelo left a comment

Upgrade TF azurerm v2 and NatGateway pubIP migration #192

Upgrade TF azurerm v2 and NatGateway pubIP migration #192

Conversation

dkistner commented Nov 16, 2020 • edited

dkistner commented Nov 16, 2020

dkistner commented Nov 16, 2020 • edited

ialidzhikov left a comment

Choose a reason for hiding this comment

ialidzhikov commented Nov 17, 2020

dkistner commented Nov 18, 2020

ialidzhikov commented Nov 19, 2020

kon-angelo commented Nov 23, 2020

ialidzhikov commented Nov 23, 2020

ialidzhikov commented Nov 23, 2020

testmachinery bot commented Nov 23, 2020 • edited

ialidzhikov commented Nov 23, 2020

ialidzhikov left a comment

Choose a reason for hiding this comment

kon-angelo left a comment

Choose a reason for hiding this comment

dkistner commented Nov 16, 2020 •

edited

dkistner commented Nov 16, 2020 •

edited

testmachinery bot commented Nov 23, 2020 •

edited