Recurring error "400 route operation in progress" when applying 3-network-hub-and-spoke

[mromascanu123]

TL;DR

This happens almost every time when deploying dev, nprod or prod. Have to plan and apply again and everything is fine . But this kind of error will ruin any pipeline deploying automatically the spokes

. . .
module.base_env.module.restricted_shared_vpc[0].module.regular_service_perimeter.google_access_context_manager_service_perimeter.regular_service_perimeter: Creating...
module.base_env.module.restricted_shared_vpc[0].module.regular_service_perimeter.google_access_context_manager_service_perimeter.regular_service_perimeter: Creation complete after 3s [id=accessPolicies/6329355927/servicePerimeters/sp_n_shared_restricted_default_perimeter_e480]
module.base_env.module.restricted_shared_vpc[0].module.regular_service_perimeter.google_access_context_manager_service_perimeter_resource.service_perimeter_resource["115822756025"]: Creating...
module.base_env.module.restricted_shared_vpc[0].module.regular_service_perimeter.google_access_context_manager_service_perimeter_resource.service_perimeter_resource["115822756025"]: Creation complete after 2s [id=accessPolicies/6329355927/servicePerimeters/sp_n_shared_restricted_default_perimeter_e480/projects/115822756025]
module.base_env.module.restricted_shared_vpc[0].google_access_context_manager_service_perimeter.bridge_to_network_hub_perimeter[0]: Creating...
module.base_env.module.restricted_shared_vpc[0].google_access_context_manager_service_perimeter.bridge_to_network_hub_perimeter[0]: Creation complete after 0s [id=accessPolicies/6329355927/servicePerimeters/spb_c_to_n_shared_restricted_bridge_e480]

Error: Error adding network peering: googleapi: Error 400: There is a route operation in progress on the local or peer network. Try again later., badRequest

with module.base_env.module.base_shared_vpc[0].module.peering[0].google_compute_network_peering.peer_network_peering,
on .terraform/modules/base_env.base_shared_vpc.peering/modules/network-peering/main.tf line 50, in resource "google_compute_network_peering" "peer_network_peering":
50: resource "google_compute_network_peering" "peer_network_peering" {

Did not investigate in detail what's going on, might be a race condition / unaccounted for dependency

Expected behavior

Should smoothly deploy - why the 2'nd time succeeds?

Observed behavior

Look at TL;DR*
Error: Error adding network peering: googleapi: Error 400: There is a route operation in progress on the local or peer network. Try again later., badRequest

Terraform Configuration

N/A - default, nothing special

Terraform Version

terraform version
Terraform v1.6.0
on linux_amd64

Additional information

No response

[obriensystems]

Reference: last full 3-networks-hub-and-spoke apply - up to 5-app-infra - TF 1.3.10 to avoid the issue running cloudbuild with 1.3 - if we use the default 1.7.5 (since downgraded in 1.5.7 in cloud shell)
Env: cloud shell and CB/CSR

• there were several workarounds and harcoding changes I did to get up to step 17 - for nonproduction and production - and I was running the 2 default US based regions.
  Canadian Public Sector Secure PBMM Landing Zone reference using Terraform 1.6 (for now 1.3.10) based on the PSO/TOC ready TEF V4 - Full clean organization deployment with mitigation/automation/parameterization modifications GoogleCloudPlatform/pbmm-on-gcp-onboarding#360 (comment)

I will also retest 3-nhas as soon as I finish the TEF upstream sync for 20240511 main in GoogleCloudPlatform/pbmm-on-gcp-onboarding#387 to reverify 3-nhas. There are 2 symlinks in nonproduction that need to be reverted un #1107 but they function with a double symlink ok for now.

[sleighton2022]

If there are too many simultaneous operations on peering, this will occur. It is not occurring in our integration tests. Are environments being deployed in parallel. One workaround is to set in Terraform parallel=1, but it will make the build take a long time, as you are not running in parallel.