feat: List objects dispatch throttling with seperate queues and configuration

[poovamraj]

Implement Dispatch Throttling for ListObjects

❗ We are renaming the configurations to be more specific and the old configurations are deprecated now and will be removed in a future release

• OPENFGA_DISPATCH_THROTTLING_ENABLED -> OPENFGA_CHECK_DISPATCH_THROTTLING_ENABLED
• OPENFGA_DISPATCH_THROTTLING_FREQUENCY -> OPENFGA_CHECK_DISPATCH_THROTTLING_FREQUENCY
• OPENFGA_DISPATCH_THROTTLING_THRESHOLD -> OPENFGA_CHECK_DISPATCH_THROTTLING_THRESHOLD
• OPENFGA_DISPATCH_THROTTLING_MAX_THRESHOLD -> OPENFGA_CHECK_DISPATCH_THROTTLING_MAX_THRESHOLD

Description

• We have abstracted dispatch throttling into it's own Throttler
• CheckResolver will use this Throttler with a delegate and resolver
• ListObjects uses throttler using the Throttle method
• We have a two throttlers which means a two queues and two configuration will be used (Seperate ones for Check and LIstObjects)

Previous PR - #1502

Review Checklist

• I have clicked on "allow edits by maintainers".
• I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
• The correct base branch is being used, if not main
• I have added tests to validate that the change in functionality is working as expected

[jon-whit]

@poovamraj we can't break config like this without an appropriate deprecation period. If we want to rename config values, that's great, but we need to roll it out in with a deprecation notice and a switch over.

[jon-whit]

I noticed --listObjects-dispatch-throttling-frequency can be greater than the server configured request timeout. The same is true of --check-dispatch-throttling-frequency. That shouldn't be allowed. It doesn't really make sense to delay processing dispatches for an amount of time greater than the server's request timeout.

[jon-whit]

Something is wrong with the implementation right now. We get inconsistent query responses depending on timing behavior. In some cases we get errors and in others we get a partial response.

Here's how to reproduce:
Run the server:

./openfga run --listObjects-dispatch-throttling-enabled=true --listObjects-dispatch-throttling-frequency=6s --listObjects-dispatch-throttling-frequency-threshold=2

1. Create a Store with an FGA model

model
  schema 1.1

type user

type group
  relations
    define member: [user, group#member]

export STORE_ID=$(fga store create --name kube-demo --model model.fga | jq -r .store.id)

2. Write some nested group to group relationship tuples

fga tuple write --store-id=$STORE_ID group:2#member member group:1
fga tuple write --store-id=$STORE_ID group:3#member member group:2
fga tuple write --store-id=$STORE_ID group:4#member member group:3

3. Call ListObjects(group#member, group:4#member)
   You get the following response:

2024-05-17T11:49:01.417-0500	INFO	grpc_req_complete	{"grpc_service": "openfga.v1.OpenFGAService", "grpc_method": "ListObjects", "grpc_type": "unary", "user_agent": "PostmanRuntime/7.38.0", "raw_request": {"store_id":"01HY3Q7DAG7KVY0MM0NF95JH5Z","authorization_model_id":"","type":"group","relation":"member","user":"group:4#member","contextual_tuples":null,"context":null}, "raw_response": {"objects":["group:4","group:3","group:2","group:1"]}, "peer.address": "127.0.0.1:59483", "request_id": "bf788e81-2bfc-4591-bb12-c2ee8a5a8051", "store_id": "01HY3Q7DAG7KVY0MM0NF95JH5Z", "authorization_model_id": "01HY3Q7DAPBNPJ72W8QF0E5MWS", "datastore_query_count": 4, "dispatch_count": 3, "grpc_code": 0}

Notice the dispatch_count is 3. The dispatch count is 3 because we had to traverse 3 indirect relationships from group:4 --> (group:3 --> group:2 --> group:1).

Sometimes we get a partial response and sometimes we surface a "deadline exceeded error". For example:

➜  ~ fga query list-objects --store-id=$STORE_ID group:4#member member group
Error: failed to list objects due to failed to list objects due to ListObjects internal error for POST ListObjects with body {"code":"deadline_exceeded","message":"context deadline exceeded"}
 with error code deadline_exceeded error message: context deadline exceeded

versus

➜  ~ fga query list-objects --store-id=$STORE_ID group:4#member member group
{
  "objects": [
    "group:4",
    "group:3"
  ]
}

It's ok for us to return a partial response, but it's not ok for us to return an error in some cases and partial responses in others. We should resolve that before we call this "complete"