Placement: disseminate min Actor API version + reject nodes with version lower than minium

[ItalyPaleAle]

Fixes #6838

Completes the work that was started in 1.12:

• The placement service collects all API versions from dapr (which were already reported starting with Dapr 1.12, but unused)
• If a node tries to join a cluster and its actor API version is lower than the minimum observed in the cluster, it's rejected
• It's possible to manually set a "ceiling" using -max-api-version when starting the placement service
• The cluster API version is disseminated to all sidecars connected
• Once a cluster reaches a given API level, it can never be downgraded (unless users pass -max-api-version). This is to prevent issues such as clusters with mixed versions, in which Placement is restarted, and the minimum version depends on the order the apps come back online at.

The behavior is validated with new ITs, as well as updated unit tests.

Note:

• Although daprd now receives the API level for the cluster, it is currently unused in Daprd - meaning, it doesn't yet use it to make any decision (a decision will be Serialize reminders as protobuf in state store #7129)

[codecov]

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (6bf645f) 65.07% compared to head (3ddc6a7) 64.98%.
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7134      +/-   ##
==========================================
- Coverage   65.07%   64.98%   -0.10%     
==========================================
  Files         221      221              
  Lines       20798    20847      +49     
==========================================
+ Hits        13534    13547      +13     
- Misses       6125     6156      +31     
- Partials     1139     1144       +5     

Files	Coverage Δ
cmd/placement/options/options.go	94.44% <100.00%> (+0.32%)	⬆️
pkg/injector/patcher/sidecar_container.go	89.82% <100.00%> (+0.03%)	⬆️
pkg/placement/raft/fsm.go	76.82% <100.00%> (+0.87%)	⬆️
pkg/placement/raft/state.go	92.38% <89.47%> (-1.03%)	⬇️
pkg/placement/membership.go	48.38% <20.00%> (-5.39%)	⬇️
pkg/placement/placement.go	73.17% <66.66%> (-4.31%)	⬇️
pkg/operator/operator.go	0.00% <0.00%> (ø)
pkg/placement/tables.go	0.00% <0.00%> (ø)

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mukundansundar]

should a minActorApiLevel also be a configurable parameter?
Do we consider the APILevel there in placement service as the min API level or should we have a separate value?
i.e. let's say that there is complete backwards compatibility between levels 10, 11 and 12 ....
So some daprd versions can join with 10 others with 11 others with 12 ... But the min level can be 10 and the cluster level can still be 11.

[mukundansundar]

is this a blocking call which takes some time, and if so should now be calculated again after the loop?

[ItalyPaleAle]

Not a blocking call. It just made sense to get the current time once only, so it's the same everywhere, down to the nanoseconds.

[ItalyPaleAle]

should a minActorApiLevel also be a configurable parameter?

I didn't think of that at the beginning, but I now can see a use case for this. I will add it.

Do we consider the APILevel there in placement service as the min API level or should we have a separate value

The API level reported by placement is the minimum observed API level in the cluster

i.e. let's say that there is complete backwards compatibility between levels 10, 11 and 12 ..

The design doc #6838 states that versions are never fully backwards-compatible. The version does not track the daprd version however, so if no change is made that requires updating the version in a Dapr release, it won't be updated.

[yaron2]

Sets the "max-api-level" flag which prevents the Actor API level for going above this value. The default value of -1 means no cap - this description isn't clear to users. lets make this very clear so users know why they're given this option, what it affects and why they care about it - otherwise this Helm chart setting needs to be removed.

[ItalyPaleAle]

Updated per review feedback. Added "min" option and clarified message on Helm chart.

[daixiang0]

LGTM

[JoshVanL]

Suggested change

	| `dapr_placement.maxApiLevel` | Sets the `max-api-level` flag which prevents the Actor API level for going above this value. The default value of -1 means no cap. | `-1` |
	| `dapr_placement.maxApiLevel` | Sets the `max-api-level` flag which prevents the Actor API level from going above this value. The default value of -1 means no cap. | `-1` |

[JoshVanL]

Could you also add a bit of blurb as to why a user would want to change this? It is not currently clear to me why a user would want to use this flag, and question why it needs to be exposed by the helm chart at all. Is this not an internal value which is hard coded into the versioned build?

[ItalyPaleAle]

How about I add the more detailed explanation in the docs? I'd rather not write a long paragraph here.

Most users will not need to set this, but it can be necessary in some rare situations.

By default this is automatically set to the minimum observed value in the cluster. For example, say you have 3 apps, with these actor API levels: 10, 20, 20. The minimum observed is 10, so that will be the level used in the cluster.

Next, assume that the app with level 10 goes away, so the minimum level now becomes 20.

The level can never be decreased, only increased. Which means that if now the app with level 10 comes back, it will not be able to connect to Placement (and use actors).

In this very specific case, users can set maxApiLevel to 10 to enforce the reported level is never higher than 10.

I can see this possibly being the case during upgrades or if users have the need to run older versions of a sidecar alongside new ones. It should not be a common case, however.