Hot Reloading: Kubernetes Component hot reloading #7260
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #7260 +/- ##
==========================================
- Coverage 64.70% 64.49% -0.21%
==========================================
Files 233 236 +3
Lines 21517 21639 +122
==========================================
+ Hits 13922 13957 +35
- Misses 6396 6482 +86
- Partials 1199 1200 +1 ☔ View full report in Codecov by Sentry. |
JoshVanL
force-pushed
the
hot-reloading-components-operator-tested
branch
2 times, most recently
from
bc1b9e4
to
4aee6ed
Compare
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Dec 11, 2023
Part of dapr#1172 Branched from dapr#7260 Adds support for HTTP middleware hot reloading. PR adds support for HTTP middlewares to be dynamically updated. The HTTP Middleware returns a middleware.HTTP pipleine built from a config spec. This pipeline implements a single handler which it itself runs the spec configured HTTP middleware chain. When a middleware Component is added or removed, the HTTP middleware store is updated and both the HTTP server and App HTTP channel's pipeline's chains are updated dynamically. Like today, if a pipeline spec contains handlers whose names, version, or type does not match that from the store, then the handler is skipped for that chain. Pipeline handler order is preserved (i.e. reverse order). The middleware store has been made generic so can be used in future for planned gRPC middleware support. Middleware init has been moved from the runtime channel manager to the runtime processor init procedure to allow for dynamic loading. The middleware HTTP manager is passed to the runtime processor to expose the store, and relevant built pipelines passed to the HTTP app channel and HTTP server. Middleware integration tests has been added for daprd to check functionality, ordering, and hotreloading, Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Dec 11, 2023
Part of dapr#1172 Branched from dapr#7260 Adds support for actor state store hot reloading. PR updates the actors package so that the actor state store is dynamically retrieved from the component store every time the state it invoked. The passed static actor state store name previously used to retrieve the state store has been removed. The component store has been updated to track the current actor state store. It is no longer tracked by the state processor, and instead the state processor updates the latest in the component store. Like before, the actor state store _must_ be configured on Dapr startup, else the actor subsystem runtime will not be enabled, even if an actor state store is hot reloaded later. Adds actor hotreloading integration tests for the actor state store. Signed-off-by: joshvanl <me@joshvanl.dev>
Part of dapr#1172 Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature. If enabled, Daprd in Selfhosted mode will watch for resource updates from watching the resource directory(s). If a resource is detected as changed (spec is different to what is currently loaded), then Dapr will close/create/update the resource. Supports Component _except_ HTTP middlware which requires further changes to the HTTP server handler machinery. A warning is thrown if hot reloading is enabled and a HTTP middleware is updated. The hot reloader reconciler is made generic to enable other resource types to be added in future. --- When running in standalone mode, the disk loader watches the directories passed by --resources-path for updates to yaml files containing Components. When an event occurs, the reconciler and differ are responsible for determining whether any resources have been created, updated, or deleted, by comparing the local store specs with that of the remote. If any have changed, the resource is closed (if it exists), and then re-initialized (if it exists). A resource will only be closed if it has been deleted, only initialized if it has been created, and closed & initialized if it has been updated. We consider a resource to have changed generally if anything apart from its Kubernetes metadata or type meta object meta has changed, and therefore needs some kind of reloading. --- Currently, if a reloading component errors and `spec.ignoreErrors=false` then Daprd will gracefully close, like a component loaded at startup today. It is intended that in future the component will be re-inited on a backoff queue in future in the case of errors, even if `spec.ignoreErros=true`. HTTP middleware component hot reloading is not supported as it requires further changes to the HTTP server handler machinery, and I didn't want to grow this PR further. To use the HotReloading feature, which is currently only available as an alpha feature in Selfhosted mode, users need to add the following Configuration to the target Daprd; ```yaml apiVersion: dapr.io/v1alpha1 kind: Configuration metadata: name: hotreloading spec: features: - name: HotReload enabled: true ``` Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
each event Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Part of dapr#1172 Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature. If enabled, Daprd in Selfhosted mode will watch for resource updates from watching the resource directory(s). If a resource is detected as changed (spec is different to what is currently loaded), then Dapr will close/create/update the resource. Supports Component _except_ HTTP middlware which requires further changes to the HTTP server handler machinery. A warning is thrown if hot reloading is enabled and a HTTP middleware is updated. The hot reloader reconciler is made generic to enable other resource types to be added in future. --- When running in standalone mode, the disk loader watches the directories passed by --resources-path for updates to yaml files containing Components. When an event occurs, the reconciler and differ are responsible for determining whether any resources have been created, updated, or deleted, by comparing the local store specs with that of the remote. If any have changed, the resource is closed (if it exists), and then re-initialized (if it exists). A resource will only be closed if it has been deleted, only initialized if it has been created, and closed & initialized if it has been updated. We consider a resource to have changed generally if anything apart from its Kubernetes metadata or type meta object meta has changed, and therefore needs some kind of reloading. --- Currently, if a reloading component errors and `spec.ignoreErrors=false` then Daprd will gracefully close, like a component loaded at startup today. It is intended that in future the component will be re-inited on a backoff queue in future in the case of errors, even if `spec.ignoreErros=true`. HTTP middleware component hot reloading is not supported as it requires further changes to the HTTP server handler machinery, and I didn't want to grow this PR further. To use the HotReloading feature, which is currently only available as an alpha feature in Selfhosted mode, users need to add the following Configuration to the target Daprd; ```yaml apiVersion: dapr.io/v1alpha1 kind: Configuration metadata: name: hotreloading spec: features: - name: HotReload enabled: true ``` Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Branched from dapr#7239 Part of dapr#1172 Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature. If enabled, Daprd in Kubernetes mode will watch for Component resource updates from the Operator. If a resource is detected as changed (spec is different to what is currently loaded), then Dapr will close/create/update the resource. Supports Component _except_ HTTP middlware which requires further changes to the HTTP server handler machinery. A warning is thrown if hot reloading is enabled and a HTTP middleware is updated. --- When running in Kubernetes mode, the operator loader streams updates from the Operator API. When an event occurs, the event type will trigger the resource to be created, updated, or deleted. Updated components will be closed, and then re-initialized. We consider a resource to have changed generally if anything apart from its Kubernetes metadata or type meta object meta has changed, and therefore needs some kind of reloading. --- Currently, if a reloading component errors and `spec.ignoreErrors=false` then Daprd will gracefully close, like a component loaded at startup today. It is intended that in future the component will be re-inited on a backoff queue in future in the case of errors, even if `spec.ignoreErros=true`. HTTP middleware component hot reloading is not supported as it requires further changes to the HTTP server handler machinery, and I didn't want to grow this PR further. To use the HotReloading feature, which is currently only available as an alpha feature in Selfhosted mode, users need to add the following Configuration to the target Daprd; ```yaml apiVersion: dapr.io/v1alpha1 kind: Configuration metadata: name: hotreloading spec: features: - name: HotReload enabled: true ``` Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Part of dapr#1172 Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature. If enabled, Daprd in Selfhosted mode will watch for resource updates from watching the resource directory(s). If a resource is detected as changed (spec is different to what is currently loaded), then Dapr will close/create/update the resource. Supports Component _except_ HTTP middlware which requires further changes to the HTTP server handler machinery. A warning is thrown if hot reloading is enabled and a HTTP middleware is updated. The hot reloader reconciler is made generic to enable other resource types to be added in future. --- When running in standalone mode, the disk loader watches the directories passed by --resources-path for updates to yaml files containing Components. When an event occurs, the reconciler and differ are responsible for determining whether any resources have been created, updated, or deleted, by comparing the local store specs with that of the remote. If any have changed, the resource is closed (if it exists), and then re-initialized (if it exists). A resource will only be closed if it has been deleted, only initialized if it has been created, and closed & initialized if it has been updated. We consider a resource to have changed generally if anything apart from its Kubernetes metadata or type meta object meta has changed, and therefore needs some kind of reloading. --- Currently, if a reloading component errors and `spec.ignoreErrors=false` then Daprd will gracefully close, like a component loaded at startup today. It is intended that in future the component will be re-inited on a backoff queue in future in the case of errors, even if `spec.ignoreErros=true`. HTTP middleware component hot reloading is not supported as it requires further changes to the HTTP server handler machinery, and I didn't want to grow this PR further. To use the HotReloading feature, which is currently only available as an alpha feature in Selfhosted mode, users need to add the following Configuration to the target Daprd; ```yaml apiVersion: dapr.io/v1alpha1 kind: Configuration metadata: name: hotreloading spec: features: - name: HotReload enabled: true ``` Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL
force-pushed
the
hot-reloading-components-operator-tested
branch
from
4aee6ed
to
f3e5d60
Compare
ItalyPaleAle
requested changes
Jan 2, 2024
JoshVanL
added
the
autoupdate
Signed-off-by: joshvanl <me@joshvanl.dev>
|
Thanks @ItalyPaleAle, done! |
Signed-off-by: joshvanl <me@joshvanl.dev>
yaron2
reviewed
Jan 3, 2024
|
|
||
| func (r *resource[T]) Stream(ctx context.Context) (<-chan *loader.Event[T], error) { | ||
| if r.closed.Load() { | ||
| return nil, errors.New("stream is closed") |
There was a problem hiding this comment.
Who is responsible to re-establish the stream if its closed?
There was a problem hiding this comment.
This is a sanity check to return early if we call Stream after the resource streamer manager has been closed (i.e. we are shutting down). Here we retry forever with an exponential backoff until the context is cancelled to reconnect to the operator:
yaron2
approved these changes
Jan 4, 2024
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Jan 9, 2024
Part of dapr#1172 Branched from dapr#7260 Adds support for HTTP middleware hot reloading. PR adds support for HTTP middlewares to be dynamically updated. The HTTP Middleware returns a middleware.HTTP pipleine built from a config spec. This pipeline implements a single handler which it itself runs the spec configured HTTP middleware chain. When a middleware Component is added or removed, the HTTP middleware store is updated and both the HTTP server and App HTTP channel's pipeline's chains are updated dynamically. Like today, if a pipeline spec contains handlers whose names, version, or type does not match that from the store, then the handler is skipped for that chain. Pipeline handler order is preserved (i.e. reverse order). The middleware store has been made generic so can be used in future for planned gRPC middleware support. Middleware init has been moved from the runtime channel manager to the runtime processor init procedure to allow for dynamic loading. The middleware HTTP manager is passed to the runtime processor to expose the store, and relevant built pipelines passed to the HTTP app channel and HTTP server. Middleware integration tests has been added for daprd to check functionality, ordering, and hotreloading, Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Jan 9, 2024
Part of dapr#1172 Branched from dapr#7260 Adds support for actor state store hot reloading. PR updates the actors package so that the actor state store is dynamically retrieved from the component store every time the state it invoked. The passed static actor state store name previously used to retrieve the state store has been removed. The component store has been updated to track the current actor state store. It is no longer tracked by the state processor, and instead the state processor updates the latest in the component store. Like before, the actor state store _must_ be configured on Dapr startup, else the actor subsystem runtime will not be enabled, even if an actor state store is hot reloaded later. Adds actor hotreloading integration tests for the actor state store. Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Jan 14, 2024
Part of dapr#1172 Branched from dapr#7260 Adds support for HTTP middleware hot reloading. PR adds support for HTTP middlewares to be dynamically updated. The HTTP Middleware returns a middleware.HTTP pipleine built from a config spec. This pipeline implements a single handler which it itself runs the spec configured HTTP middleware chain. When a middleware Component is added or removed, the HTTP middleware store is updated and both the HTTP server and App HTTP channel's pipeline's chains are updated dynamically. Like today, if a pipeline spec contains handlers whose names, version, or type does not match that from the store, then the handler is skipped for that chain. Pipeline handler order is preserved (i.e. reverse order). The middleware store has been made generic so can be used in future for planned gRPC middleware support. Middleware init has been moved from the runtime channel manager to the runtime processor init procedure to allow for dynamic loading. The middleware HTTP manager is passed to the runtime processor to expose the store, and relevant built pipelines passed to the HTTP app channel and HTTP server. Middleware integration tests has been added for daprd to check functionality, ordering, and hotreloading, Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Jan 18, 2024
Part of dapr#1172 Branched from dapr#7260 Adds support for HTTP middleware hot reloading. PR adds support for HTTP middlewares to be dynamically updated. The HTTP Middleware returns a middleware.HTTP pipleine built from a config spec. This pipeline implements a single handler which it itself runs the spec configured HTTP middleware chain. When a middleware Component is added or removed, the HTTP middleware store is updated and both the HTTP server and App HTTP channel's pipeline's chains are updated dynamically. Like today, if a pipeline spec contains handlers whose names, version, or type does not match that from the store, then the handler is skipped for that chain. Pipeline handler order is preserved (i.e. reverse order). The middleware store has been made generic so can be used in future for planned gRPC middleware support. Middleware init has been moved from the runtime channel manager to the runtime processor init procedure to allow for dynamic loading. The middleware HTTP manager is passed to the runtime processor to expose the store, and relevant built pipelines passed to the HTTP app channel and HTTP server. Middleware integration tests has been added for daprd to check functionality, ordering, and hotreloading, Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL
added a commit
to JoshVanL/dapr
that referenced
this pull request
Jan 25, 2024
Part of dapr#1172 Branched from dapr#7260 Adds support for HTTP middleware hot reloading. PR adds support for HTTP middlewares to be dynamically updated. The HTTP Middleware returns a middleware.HTTP pipleine built from a config spec. This pipeline implements a single handler which it itself runs the spec configured HTTP middleware chain. When a middleware Component is added or removed, the HTTP middleware store is updated and both the HTTP server and App HTTP channel's pipeline's chains are updated dynamically. Like today, if a pipeline spec contains handlers whose names, version, or type does not match that from the store, then the handler is skipped for that chain. Pipeline handler order is preserved (i.e. reverse order). The middleware store has been made generic so can be used in future for planned gRPC middleware support. Middleware init has been moved from the runtime channel manager to the runtime processor init procedure to allow for dynamic loading. The middleware HTTP manager is passed to the runtime processor to expose the store, and relevant built pipelines passed to the HTTP app channel and HTTP server. Middleware integration tests has been added for daprd to check functionality, ordering, and hotreloading, Signed-off-by: joshvanl <me@joshvanl.dev>
artursouza
pushed a commit
that referenced
this pull request
Jan 26, 2024
* Hot Reloading: HTTP Middleware Reloading Part of #1172 Branched from #7260 Adds support for HTTP middleware hot reloading. PR adds support for HTTP middlewares to be dynamically updated. The HTTP Middleware returns a middleware.HTTP pipleine built from a config spec. This pipeline implements a single handler which it itself runs the spec configured HTTP middleware chain. When a middleware Component is added or removed, the HTTP middleware store is updated and both the HTTP server and App HTTP channel's pipeline's chains are updated dynamically. Like today, if a pipeline spec contains handlers whose names, version, or type does not match that from the store, then the handler is skipped for that chain. Pipeline handler order is preserved (i.e. reverse order). The middleware store has been made generic so can be used in future for planned gRPC middleware support. Middleware init has been moved from the runtime channel manager to the runtime processor init procedure to allow for dynamic loading. The middleware HTTP manager is passed to the runtime processor to expose the store, and relevant built pipelines passed to the HTTP app channel and HTTP server. Middleware integration tests has been added for daprd to check functionality, ordering, and hotreloading, Signed-off-by: joshvanl <me@joshvanl.dev> * Update pkg/middleware/http/http.go Co-authored-by: Loong Dai <long0dai@foxmail.com> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> * Remove false comment Signed-off-by: joshvanl <me@joshvanl.dev> * Update middleware tests due to new reserved dapr Component name Signed-off-by: joshvanl <me@joshvanl.dev> * Fix http channel test panic Signed-off-by: joshvanl <me@joshvanl.dev> * Remove workflow.dapr component from middleware integration test metadata component checks Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: Loong Dai <long0dai@foxmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Branched from #7239
Part of #1172
Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.
If enabled, Daprd in Kubernetes mode will watch for Component resource
updates from the Operator. If a resource is detected as changed (spec is
different to what is currently loaded), then Dapr will
close/create/update the resource.
Supports Component except HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.
When running in Kubernetes mode, the operator loader streams updates from
the Operator API. When an event occurs, the event type will trigger the
resource to be created, updated, or deleted. Updated components will be
closed, and then re-initialized.
We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.
Currently, if a reloading component errors and
spec.ignoreErrors=falsethen Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
spec.ignoreErros=true.HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.
To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;