Rate-limited Hydra APIs cause breakage

[tjorri]

We've encountered some issues when interacting with a larger number of OAuth2 clients using the provider. Specifically we're seeing the following:

1. The errors.As in readOAuth2ClientResource panics with a non-nil pointer.
2. Hydra APIs can be rate-limited (especially for Ory Network provided Hydra, https://www.ory.sh/docs/guides/rate-limits), so an optional mechanism for backing off and retrying HTTP 429 responses from Hydra SDK calls is introduced.

Both issues can be reproduced by creating a sufficient amount of Hydra OAuth2 clients[1] (e.g. 50) against a rate-limited Hydra and then refreshing the state. With the existing provider version, this should cause an error, which causes the panic[2]. After fixing the error.As issue, we then hit the 429 throttling[2], for which the only immediate recourse without touching the provider is setting Terraform parallelism to something low, which reduces the pressure on the API and avoids getting throttled, but which also then makes certain Terraform runs prohibitively length.

I've provided an example of solving these in pull request #20.

[1] Sample Terraform setup to repro with:

terraform {
  required_providers {
    hydra = {
      source = "svrakitin/hydra"
      version = "0.5.2"
    }
  }
}

provider "hydra" {
  endpoint = "https://PROJECT-SLUG.projects.oryapis.com"

  authentication {
    http_header {
      name  = "authorization"
      value = "bearer ORY_TOKEN"
    }
  }
}

resource "hydra_oauth2_client" "example" {
  count = 50

  client_id   = "test-client-${count.index}"
  client_name = "test-client-${count.index}"

  redirect_uris = ["http://localhost:8080/callback"]

  response_types             = ["code"]
  token_endpoint_auth_method = "none"
}

[2] Example of provider panic when refreshing:

% terraform apply
hydra_oauth2_client.example[19]: Refreshing state... [id=test-client-19]
hydra_oauth2_client.example[17]: Refreshing state... [id=test-client-17]
hydra_oauth2_client.example[25]: Refreshing state... [id=test-client-25]
hydra_oauth2_client.example[3]: Refreshing state... [id=test-client-3]
hydra_oauth2_client.example[42]: Refreshing state... [id=test-client-42]
hydra_oauth2_client.example[8]: Refreshing state... [id=test-client-8]
hydra_oauth2_client.example[1]: Refreshing state... [id=test-client-1]
hydra_oauth2_client.example[29]: Refreshing state... [id=test-client-29]
hydra_oauth2_client.example[5]: Refreshing state... [id=test-client-5]
hydra_oauth2_client.example[40]: Refreshing state... [id=test-client-40]
hydra_oauth2_client.example[35]: Refreshing state... [id=test-client-35]
hydra_oauth2_client.example[23]: Refreshing state... [id=test-client-23]
hydra_oauth2_client.example[2]: Refreshing state... [id=test-client-2]
hydra_oauth2_client.example[27]: Refreshing state... [id=test-client-27]
hydra_oauth2_client.example[22]: Refreshing state... [id=test-client-22]
hydra_oauth2_client.example[13]: Refreshing state... [id=test-client-13]
hydra_oauth2_client.example[28]: Refreshing state... [id=test-client-28]
hydra_oauth2_client.example[46]: Refreshing state... [id=test-client-46]
hydra_oauth2_client.example[31]: Refreshing state... [id=test-client-31]
hydra_oauth2_client.example[0]: Refreshing state... [id=test-client-0]
hydra_oauth2_client.example[4]: Refreshing state... [id=test-client-4]
hydra_oauth2_client.example[14]: Refreshing state... [id=test-client-14]
hydra_oauth2_client.example[48]: Refreshing state... [id=test-client-48]
hydra_oauth2_client.example[26]: Refreshing state... [id=test-client-26]
hydra_oauth2_client.example[21]: Refreshing state... [id=test-client-21]
hydra_oauth2_client.example[38]: Refreshing state... [id=test-client-38]
hydra_oauth2_client.example[49]: Refreshing state... [id=test-client-49]
hydra_oauth2_client.example[39]: Refreshing state... [id=test-client-39]
hydra_oauth2_client.example[11]: Refreshing state... [id=test-client-11]
hydra_oauth2_client.example[33]: Refreshing state... [id=test-client-33]
hydra_oauth2_client.example[47]: Refreshing state... [id=test-client-47]
hydra_oauth2_client.example[18]: Refreshing state... [id=test-client-18]
hydra_oauth2_client.example[32]: Refreshing state... [id=test-client-32]
hydra_oauth2_client.example[12]: Refreshing state... [id=test-client-12]
hydra_oauth2_client.example[7]: Refreshing state... [id=test-client-7]
hydra_oauth2_client.example[37]: Refreshing state... [id=test-client-37]
hydra_oauth2_client.example[44]: Refreshing state... [id=test-client-44]
hydra_oauth2_client.example[15]: Refreshing state... [id=test-client-15]
hydra_oauth2_client.example[20]: Refreshing state... [id=test-client-20]
hydra_oauth2_client.example[45]: Refreshing state... [id=test-client-45]
hydra_oauth2_client.example[41]: Refreshing state... [id=test-client-41]
hydra_oauth2_client.example[10]: Refreshing state... [id=test-client-10]
hydra_oauth2_client.example[34]: Refreshing state... [id=test-client-34]
hydra_oauth2_client.example[36]: Refreshing state... [id=test-client-36]
hydra_oauth2_client.example[24]: Refreshing state... [id=test-client-24]
hydra_oauth2_client.example[9]: Refreshing state... [id=test-client-9]
hydra_oauth2_client.example[43]: Refreshing state... [id=test-client-43]
hydra_oauth2_client.example[6]: Refreshing state... [id=test-client-6]
hydra_oauth2_client.example[30]: Refreshing state... [id=test-client-30]
hydra_oauth2_client.example[16]: Refreshing state... [id=test-client-16]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[6],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[24],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[43],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[16],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[34],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[9],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[30],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│   with hydra_oauth2_client.example[36],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The
│ plugin logs may contain more details.
╵

Stack trace from the terraform-provider-hydra_v0.5.2 plugin:

panic: errors: target must be a non-nil pointer

goroutine 138 [running]:
errors.As({0x1053c5cd8, 0x1400052f700}, {0x1052f36c0?, 0x0})
	errors/wrap.go:103 +0x470
github.com/svrakitin/terraform-provider-hydra/internal/provider.readOAuth2ClientResource({0x1053cccb0, 0x1400062eee0}, 0x14000510c80, {0x105307720?, 0x1400050f260})
	github.com/svrakitin/terraform-provider-hydra/internal/provider/resource_oauth2_client.go:343 +0x188
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x140000e4c40, {0x1053ccc08, 0x14000739620}, 0xd?, {0x105307720, 0x1400050f260})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:795 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0x140000e4c40, {0x1053ccc08, 0x14000739620}, 0x14000a8cea0, {0x105307720, 0x1400050f260})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:1089 +0x430
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x14000371bc0, {0x1053ccc08?, 0x140007dfef0?}, 0x1400052eb00)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/grpc_provider.go:649 +0x3e4
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x14000255b80, {0x1053ccc08?, 0x140007df710?}, 0x140003b3560)
	github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:789 +0x390
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x105392aa0?, 0x14000255b80}, {0x1053ccc08, 0x140007df710}, 0x14000510980, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:431 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x1400024a1e0, {0x1053ccc08, 0x140007df5f0}, {0x1053d0db8, 0x1400025d860}, 0x14000566120, 0x14000383980, 0x1058ed9a8, 0x0)
	google.golang.org/grpc@v1.59.0/server.go:1343 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x1400024a1e0, {0x1053d0db8, 0x1400025d860}, 0x14000566120)
	google.golang.org/grpc@v1.59.0/server.go:1737 +0x990
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc@v1.59.0/server.go:986 +0x88
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 13
	google.golang.org/grpc@v1.59.0/server.go:997 +0x160

Error: The terraform-provider-hydra_v0.5.2 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[3] Example of 429 rate-limiting:

% terraform apply
╷
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│  - svrakitin/hydra in /Users/ttj/projects/metaplay/terraform-provider-hydra
│
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state
│ to become incompatible with published releases.
╵
hydra_oauth2_client.example[27]: Refreshing state... [id=test-client-27]
hydra_oauth2_client.example[39]: Refreshing state... [id=test-client-39]
hydra_oauth2_client.example[6]: Refreshing state... [id=test-client-6]
hydra_oauth2_client.example[38]: Refreshing state... [id=test-client-38]
hydra_oauth2_client.example[19]: Refreshing state... [id=test-client-19]
hydra_oauth2_client.example[17]: Refreshing state... [id=test-client-17]
hydra_oauth2_client.example[32]: Refreshing state... [id=test-client-32]
hydra_oauth2_client.example[46]: Refreshing state... [id=test-client-46]
hydra_oauth2_client.example[22]: Refreshing state... [id=test-client-22]
hydra_oauth2_client.example[48]: Refreshing state... [id=test-client-48]
hydra_oauth2_client.example[33]: Refreshing state... [id=test-client-33]
hydra_oauth2_client.example[34]: Refreshing state... [id=test-client-34]
hydra_oauth2_client.example[18]: Refreshing state... [id=test-client-18]
hydra_oauth2_client.example[12]: Refreshing state... [id=test-client-12]
hydra_oauth2_client.example[45]: Refreshing state... [id=test-client-45]
hydra_oauth2_client.example[49]: Refreshing state... [id=test-client-49]
hydra_oauth2_client.example[20]: Refreshing state... [id=test-client-20]
hydra_oauth2_client.example[37]: Refreshing state... [id=test-client-37]
hydra_oauth2_client.example[5]: Refreshing state... [id=test-client-5]
hydra_oauth2_client.example[9]: Refreshing state... [id=test-client-9]
hydra_oauth2_client.example[29]: Refreshing state... [id=test-client-29]
hydra_oauth2_client.example[0]: Refreshing state... [id=test-client-0]
hydra_oauth2_client.example[47]: Refreshing state... [id=test-client-47]
hydra_oauth2_client.example[42]: Refreshing state... [id=test-client-42]
hydra_oauth2_client.example[25]: Refreshing state... [id=test-client-25]
hydra_oauth2_client.example[2]: Refreshing state... [id=test-client-2]
hydra_oauth2_client.example[4]: Refreshing state... [id=test-client-4]
hydra_oauth2_client.example[13]: Refreshing state... [id=test-client-13]
hydra_oauth2_client.example[23]: Refreshing state... [id=test-client-23]
hydra_oauth2_client.example[40]: Refreshing state... [id=test-client-40]
hydra_oauth2_client.example[8]: Refreshing state... [id=test-client-8]
hydra_oauth2_client.example[43]: Refreshing state... [id=test-client-43]
hydra_oauth2_client.example[31]: Refreshing state... [id=test-client-31]
hydra_oauth2_client.example[10]: Refreshing state... [id=test-client-10]
hydra_oauth2_client.example[36]: Refreshing state... [id=test-client-36]
hydra_oauth2_client.example[44]: Refreshing state... [id=test-client-44]
hydra_oauth2_client.example[28]: Refreshing state... [id=test-client-28]
hydra_oauth2_client.example[11]: Refreshing state... [id=test-client-11]
hydra_oauth2_client.example[26]: Refreshing state... [id=test-client-26]
hydra_oauth2_client.example[7]: Refreshing state... [id=test-client-7]
hydra_oauth2_client.example[14]: Refreshing state... [id=test-client-14]
hydra_oauth2_client.example[16]: Refreshing state... [id=test-client-16]
hydra_oauth2_client.example[15]: Refreshing state... [id=test-client-15]
hydra_oauth2_client.example[24]: Refreshing state... [id=test-client-24]
hydra_oauth2_client.example[1]: Refreshing state... [id=test-client-1]
hydra_oauth2_client.example[30]: Refreshing state... [id=test-client-30]
hydra_oauth2_client.example[21]: Refreshing state... [id=test-client-21]
hydra_oauth2_client.example[41]: Refreshing state... [id=test-client-41]
hydra_oauth2_client.example[35]: Refreshing state... [id=test-client-35]
hydra_oauth2_client.example[3]: Refreshing state... [id=test-client-3]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[14],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[16],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[35],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[30],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[41],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[3],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[1],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[24],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[15],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵
╷
│ Error: 429 Too Many Requests
│
│   with hydra_oauth2_client.example[21],
│   on test.tf line 21, in resource "hydra_oauth2_client" "example":
│   21: resource "hydra_oauth2_client" "example" {
│
╵