New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A68: Random subsetting with rendezvous hashing LB policy #423
base: master
Are you sure you want to change the base?
Changes from 11 commits
97c8e73
1058d36
ea7db4b
4c2ca73
c1b1b6e
8fd9fff
d58f609
5ce1497
802fde1
31cdd68
bcd293c
5c859ca
b710a6f
8796292
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
A68: Random subsetting with rendezvous hashing LB policy. | ||
---- | ||
* Author(s): @s-matyukevich | ||
* Approver: | ||
* Status: Draft | ||
* Implemented in: PoC in Go | ||
* Last updated: 2024-04-15 | ||
* Discussion at: https://groups.google.com/g/grpc-io/c/oxNJT1GgjEg | ||
|
||
## Abstract | ||
|
||
Add support for the `random_subsetting` load balancing policy. | ||
|
||
## Background | ||
|
||
Currently, gRPC is lacking a way to select a subset of endpoints available from the resolver and load-balance requests between them. Out of the box, users have the choice between two extremes: `pick_first` which sends all requests to one random backend, and `round_robin` which sends requests to all available backends. `pick_first` has poor connection balancing when the number of client is not much higher than the number of servers. The problem is exacerbated during rollouts because `pick_first` does not change endpoint on resolver updates if the current subchannel remains `READY`. `round_robin` results in every server having as many connections open as there are clients, which is unnecessarily costly when there are many clients, and makes local load balancing decisions (such as outlier detection) less precise. | ||
|
||
### Related Proposals: | ||
* [gRFC A27: A52: gRPC xDS Custom Load Balancer Configuration](https://github.com/grpc/proposal/blob/master/A52-xds-custom-lb-policies.md) | ||
|
||
## Proposal | ||
|
||
Introduce a new LB policy, `random_subsetting`. This policy selects a subset of addresses and passes them to the child LB policy. It maintains 2 important properties: | ||
* The policy tries to distribute connections among servers as equally as possible. The higher `(N_clients*subset_size)/N_servers` ratio is, the closer the resulting server connection distribution is to uniform. | ||
* The policy minimizes the amount of connection churn generated during server scale-ups by using [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing) | ||
|
||
### Subsetting algorithm | ||
|
||
* The policy receives a single configuration parameter: `subset_size`, which must be configured by the user. | ||
* When the lb policy is initialized it also creates a random 32-byte long `salt` string. | ||
* After every resolver update the policy picks a new subset. It does this by implementing `rendezvous hashing` algorithm: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would help to define the algorithm in pseudo code. See https://github.com/grpc/proposal/blob/master/A55-xds-stateful-session-affinity.md#lb-policy-for-stateful-session-affinity or https://github.com/grpc/proposal/blob/master/A42-xds-ring-hash-lb-policy.md#aggregated-connectivity-state for examples of gRFCs that do this. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
* Concatenate `salt` to each address in the list. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You'll have to decide which address, in case the endpoint has more than one (I think you can use the first address?). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated the doc to use the first address. |
||
* For every resulting entity compute [MurmurHash3](https://en.wikipedia.org/wiki/MurmurHash) hash, which produces 128-byte output. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is no dependency on murmur from grpc, at least in Go, as of today. You can use xxhash which is depended upon by There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. updated to use xxhash, This changes the algorithms slightly as we can use random pre-generated seed instead of concatenating salt to each address. The new version is even simpler. |
||
* Sort all entities by hash. | ||
* Pick first `subset_size` values from the list. | ||
* Pass the resulting subset to the child LB policy. | ||
* If the number of addresses is less than `subset_size` always use all available addresses. | ||
|
||
### Characteristics of the selected algorithm | ||
|
||
#### Uniform connection distribution on high scale | ||
|
||
When `(N_clients*subset_size)/N_servers` ratio is high, the resulting connection distribution between servers is close to uniform. This is because the chosen hash function is uniform and every server has equal probability to be chosen by every client. | ||
Though it could be done, we don't provide any mathematical guaranties about the resulting connection distribution. Any such guarantees will be probabilistic and have limited value in practice. Instead, we can give you a few samples: | ||
* N_clients = 100, N_servers = 100, subset_size = 5 | ||
![State Diagram](A68_graphics/subsetting100-100-5.png) | ||
* N_clients = 100, N_servers = 100, subset_size = 25 | ||
![State Diagram](A68_graphics/subsetting100-100-25.png) | ||
* N_clients = 100, N_servers = 10, subset_size = 5 | ||
![State Diagram](A68_graphics/subsetting100-10-5.png) | ||
* N_clients = 500, N_servers = 10, subset_size = 5 | ||
![State Diagram](A68_graphics/subsetting500-10-5.png) | ||
* N_clients = 2000, N_servers = 10, subset_size = 5 | ||
![State Diagram](A68_graphics/subsetting2000-10-5.png) | ||
|
||
#### Low connection churn during server rollouts | ||
|
||
The graphs provided in the previous section prove this is the case in practice (we rollout all servers in the middle of every test, and there is no visible increase in the number of connections per server) Low connection churn during server rollouts is the primary motivation why rendezvous hashing was used as the subsetting algorithm: it guaranties that if a single server is either added or removed to the IP address list, every client will update at most 1 entry in its subset. This is because the hashes for all unaffected servers remain the same, which guarantees that the order of the servers after sorting also remains stable. The same logic applies to the situation when multiple servers got updated. | ||
|
||
### LB Policy Config and Parameters | ||
|
||
The `random_subsetting` LB policy config will be as follows. | ||
|
||
```proto | ||
message LoadBalancingConfig { | ||
oneof policy { | ||
RandomSubsettingLbConfig random_subsetting = 21 [json_name = "random_subsetting"]; | ||
} | ||
} | ||
message RandomSubsettingLbConfig { | ||
// subset_size indicates how many backends every client will be connected to. | ||
// Default is 20. | ||
google.protobuf.UInt32Value subset_size = 1; | ||
|
||
// The config for the child policy. | ||
repeated LoadBalancingConfig child_policy = 2; | ||
} | ||
``` | ||
|
||
|
||
### Handling Parent/Resolver Updates | ||
|
||
When the resolver updates the list of addresses, or the LB config changes, Random subsetting LB will run the subsetting algorithm, described above, to filter the endpoint list. Then it will create a new resolver state with the filtered list of the addresses and pass it to the child LB. Attributes and service config from the old resolver state will be copied to the new one. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think you should replace addresses with endpoints to take A61 into consideration. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
|
||
### Handling Subchannel Connectivity State Notifications | ||
|
||
Random subsetting LB will simply redirect all requests to the child LB without doing any additional processing. This also applies to all other callbacks in the LB interface, besides the one that handles resolver and config updates (which is described in the previous section). This is possible because random subsetting LB doesn't store or manage sub-connections - it acts as a simple filter on the resolver state, and that's why it can redirect all actual work to the child LB. | ||
|
||
### xDS Integration | ||
|
||
Random subsetting LB won't depend on xDS in any way. People may choose to initialize it by directly providing service config. We will only provide a corresponding xDS policy wrapper to allow configuring this LB via xDS. | ||
|
||
#### Changes to xDS API | ||
|
||
`random_subsetting` will be added as a new LB policy. | ||
|
||
```proto | ||
package envoy.extensions.load_balancing_policies.random_subsetting.v3; | ||
message RandomSubsetting { | ||
google.protobuf.UInt32Value subset_size = 1; | ||
repeated LoadBalancingConfig child_policy = 2; | ||
} | ||
``` | ||
|
||
As you can see, the fields in this policy match exactly the fields in the random subsetting LB service config. | ||
|
||
#### Integration with xDS LB Policy Registry | ||
As described in [gRFC A52](https://github.com/grpc/proposal/blob/master/A52-xds-custom-lb-policies.md), gRPC has an LB policy registry, which maintains a list of converters. Every converter translates xDS LB policy to the corresponding service config. In order to allow using the Random subsetting LB policy via xDS, the only thing that needs to be done is providing a corresponding converter function. The function implementation will be trivial as the fields in the xDS LB policy will match exactly the fields in the service config. | ||
|
||
## Rationale | ||
### Alternatives Considered: Deterministic subsetting | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You should probably discuss the trade offs of doing this kind of subsetting in the control plane, since it was discussed in the original proposal. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, but I posted a link to the tl;dr; of the discussion, so you think this is not enough? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm thinking of the option of doing random subsetting in the control plane by sending different EDS responses (with different subsets) to each dataplane, or the equivalent with other resolvers. It is simple to implement with xDS and works for Envoy and gRPC. IIRC the main argument for not going that route is the need to have an xDS infrastructure (this is a big barrier for our orgs, and probably others), and existing limitations of https://github.com/envoyproxy/go-control-plane. This was discussed in https://github.com/grpc/proposal/pull/383/files#r1308024474. In order to understand this proposal, I think users will need to understand the trade off of doing it as a balancer in each data plane rather than directly in service discovery. |
||
|
||
We explored the possibility of using deterministic subsetting in https://github.com/grpc/proposal/pull/383 and got push-back on this for the reasons explained [here](https://github.com/grpc/proposal/pull/383#discussion_r1334587561) | ||
|
||
## Implementation | ||
DataDog will provide Go and Java implementations. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you need to replace
addresses
withendpoints
to account for https://github.com/grpc/proposal/blob/master/A61-IPv4-IPv6-dualstack-backends.md, where each endpoint may have multiple addresses.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.