[bug] Level 0 Launchpad can't swap group owners on second time around

[DevopsMercenary]

Describe the bug
A clear and concise description of what the bug is.

Performing the steps in configuration/level0/launchpad/readme.md and this is the second time where the plan and apply for the rover is like

rover \
  --impersonate-sp-from-keyvault-url https://caf-kv-idl0-ato.vault.azure.net/ \
  -lz /tf/caf/landingzones/caf_launchpad \
  -var-folder /tf/caf/configuration/level0/launchpad \
  -tfstate_subscription_id #####-18f0-45f6-abf6-3bcd9058d3eb \
  -target_subscription 0f#####-18f0-45f6-abf6-3bcd9058d3eb \
  -tfstate caf_launchpad.tfstate \
  -launchpad \
  -env myapp \
  -level level0 \
  -p ${TF_DATA_DIR}/caf_launchpad.tfstate.tfplan \
  -a plan

So the plan now shows what's remaining is to remove me as the owner and set the service principal as the new group owner.

Terraform will perform the following actions:

  # module.launchpad.module.azuread_groups["alz"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "94223cda-17f6-49ce-8855-5dbc5816ab7b"
        name                    = "caf-caf-alz"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["caf_platform_contributors"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "bdebf9a6-e375-418d-9453-1bd8a474d770"
        name                    = "caf-caf-platform-contributors"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (7 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["caf_platform_maintainers"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "b3471f35-c550-48e9-b69b-73f2d67285eb"
        name                    = "caf-caf-platform-maintainers"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (7 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["connectivity"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "38df4a87-c279-4506-9c5f-4bd8a2e9fd32"
        name                    = "caf-caf-connectivity"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["identity"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "162e40c8-79a5-4d25-b2a9-5a4a0859094b"
        name                    = "caf-caf-identity"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["level0"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "aeb3e1f9-a61c-4bfc-927c-2211eab363f5"
        name                    = "caf-caf-level0"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["management"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "b858b073-0886-46d3-bf20-6e7561b3c4cd"
        name                    = "caf-caf-management"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["subscription_creation_landingzones"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "4a14a515-e5d5-4900-b457-4f22fa5568cf"
        name                    = "caf-caf-subscription_creation_landingzones"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

  # module.launchpad.module.azuread_groups["subscription_creation_platform"].azuread_group.group will be updated in-place
  ~ resource "azuread_group" "group" {
        id                      = "9833f127-ca5c-469e-9b73-cb0258e8af69"
        name                    = "caf-caf-subscription_creation_platform"
      ~ owners                  = [
          - "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f",
            # (1 unchanged element hidden)
        ]
        # (6 unchanged attributes hidden)
    }

BUT then the apply call

rover \
  --impersonate-sp-from-keyvault-url https://caf-kv-idl0-ato.vault.azure.net/ \
  -lz /tf/caf/landingzones/caf_launchpad \
  -var-folder /tf/caf/configuration/level0/launchpad \
  -tfstate_subscription_id 0f#####-18f0-45f6-abf6-3bcd9058d3eb \
  -target_subscription 0f####-18f0-45f6-abf6-3bcd9058d3eb \
  -tfstate caf_launchpad.tfstate \
  -launchpad \
  -env myapp \
  -level level0 \
  -p ${TF_DATA_DIR}/caf_launchpad.tfstate.tfplan \
  -a apply

Can't make the swap, appearing that in terraform it tries to delete the owner (me) before adding the new one ( service principal )

module.launchpad.module.azuread_groups["alz"].azuread_group.group: Modifying... [id=94223cda-17f6-49ce-8855-5dbc5816ab7b]
module.launchpad.module.azuread_groups["level0"].azuread_group.group: Modifying... [id=aeb3e1f9-a61c-4bfc-927c-2211eab363f5]
module.launchpad.module.azuread_groups["identity"].azuread_group.group: Modifying... [id=162e40c8-79a5-4d25-b2a9-5a4a0859094b]
module.launchpad.module.azuread_groups["caf_platform_maintainers"].azuread_group.group: Modifying... [id=b3471f35-c550-48e9-b69b-73f2d67285eb]
module.launchpad.module.azuread_groups["connectivity"].azuread_group.group: Modifying... [id=38df4a87-c279-4506-9c5f-4bd8a2e9fd32]
module.launchpad.module.azuread_groups["management"].azuread_group.group: Modifying... [id=b858b073-0886-46d3-bf20-6e7561b3c4cd]
module.launchpad.module.azuread_groups["subscription_creation_landingzones"].azuread_group.group: Modifying... [id=4a14a515-e5d5-4900-b457-4f22fa5568cf]
module.launchpad.module.azuread_groups["caf_platform_contributors"].azuread_group.group: Modifying... [id=bdebf9a6-e375-418d-9453-1bd8a474d770]
module.launchpad.module.azuread_groups["subscription_creation_platform"].azuread_group.group: Modifying... [id=9833f127-ca5c-469e-9b73-cb0258e8af69]
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "38df4a87-c279-4506-9c5f-4bd8a2e9fd32"
│ 
│   with module.launchpad.module.azuread_groups["connectivity"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"c2a4c973-c093-4724-a192-0d36a8c3cb3c"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "9833f127-ca5c-469e-9b73-cb0258e8af69"
│ 
│   with module.launchpad.module.azuread_groups["subscription_creation_platform"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"361d9a3e-ac8f-424b-aa22-c57f54925599"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "b858b073-0886-46d3-bf20-6e7561b3c4cd"
│ 
│   with module.launchpad.module.azuread_groups["management"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"4bff9b13-ee10-4067-9e51-0a67a3f33393"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "94223cda-17f6-49ce-8855-5dbc5816ab7b"
│ 
│   with module.launchpad.module.azuread_groups["alz"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"046c7658-4a79-4b15-ae38-feb60f6e5bf5"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "162e40c8-79a5-4d25-b2a9-5a4a0859094b"
│ 
│   with module.launchpad.module.azuread_groups["identity"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"147f08e1-72cf-461f-99a6-69654d278ec5"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "aeb3e1f9-a61c-4bfc-927c-2211eab363f5"
│ 
│   with module.launchpad.module.azuread_groups["level0"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"8df67bf8-81d4-4149-ae11-a57a4f039826"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "b3471f35-c550-48e9-b69b-73f2d67285eb"
│ 
│   with module.launchpad.module.azuread_groups["caf_platform_maintainers"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"331eea7d-b599-4224-a023-6572ad8e768c"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "4a14a515-e5d5-4900-b457-4f22fa5568cf"
│ 
│   with module.launchpad.module.azuread_groups["subscription_creation_landingzones"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"1884bb45-97ff-4822-9dff-fa9af97bc250"}}]
╵
╷
│ Error: Removing group owner "6d639ba5-3aa6-4f5d-91d6-c8645d9a3b3f" from group with object ID: "bdebf9a6-e375-418d-9453-1bd8a474d770"
│ 
│   with module.launchpad.module.azuread_groups["caf_platform_contributors"].azuread_group.group,
│   on /home/vscode/.terraform.cache/myapp/modules/launchpad/modules/azuread/groups/group.tf line 1, in resource "azuread_group" "group":
│    1: resource "azuread_group" "group" {
│ 
│ graphrbac.GroupsClient#RemoveOwner: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown
│ service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2022-10-05T20:44:34","message":{"lang":"en","value":"The group must have at least one owner, hence this owner
│ cannot be removed."},"requestId":"d0f6fd54-4fa0-43e1-a230-d2d8cf14432b"}}]
╵
Terraform apply return code: 1

I have manually added the SP to those groups as an owner but even still I can't remove me as an owner. In some ways this looks more like an Azure issue. For instance in my console still reports one owner.

but here one the overview

[nusrath432]

@DevopsMercenary Try adding it in two steps:
say owner is user1, then add owner as user1, user2 and next step remove user1 so that user2 become the owner.