Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hot reloading of components #1172

Open
amanbha opened this issue Feb 28, 2020 · 20 comments
Open

Hot reloading of components #1172

amanbha opened this issue Feb 28, 2020 · 20 comments
Assignees
Labels
area/runtime P0 platform/kubernetes size/M 2 weeks of work triaged/resolved Indicates that this issue has been triaged

Comments

@amanbha
Copy link
Contributor

amanbha commented Feb 28, 2020

Currently in k8s, Dapr needs a pod restart to pick new configuration after they have been applied.
This will cause inconsistency in scenarios in which few pods of an app are restarted by platform. Restarted pods will use new config whereas pods which are not restarted will continue to use old config.

Tasks:

@amanbha amanbha changed the title Dapr should need a pod restart when a component is applied Dapr should not need a pod restart when a component is applied Feb 28, 2020
@artursouza artursouza added the kind/bug Something isn't working label Feb 28, 2020
@msfussell msfussell added P1 and removed kind/bug Something isn't working labels Apr 7, 2020
@msfussell
Copy link
Member

There is a discussion here on the overlap between dynamic updating of components/config by dapr and when the K8s operator manages the rollout of a new application. We need to get options from both sides for this proposal.

@msfussell msfussell changed the title Dapr should not need a pod restart when a component is applied Dapr should not need a pod restart when a component/config is applied Apr 7, 2020
@msfussell msfussell changed the title Dapr should not need a pod restart when a component/config is applied Dapr side-car should not need a restart when a component/config is applied Apr 7, 2020
@amanbha
Copy link
Contributor Author

amanbha commented Jul 21, 2020

Adding notes from maintainers meeting:
Pros and cons of dynamic updating the components::

  1. dynamic updating of components will solve a "split-brain" problem due to app scale out, failover.
  2. dynamic updating of components can also lead to issues as application code might not be ready for the change.

Suggested approaches:

  1. Keep the current behavior and document best practise that after applying the new component yaml, user should do upgrade of the application (or do pod restarts) to pick up updated components.
  2. Do dynamic upload but give user an option to opt out as an annotation.
  3. Add a dynamic upload property to each component, dapr sidecar will do "hot-reload" of the component when it receives the notification from operator only when this property is set to true.

@AaronCrawfis
Copy link
Contributor

Proposal

Given the above what it we:

  1. Keep current behavior as default (pods do not pickup changes to components until restarted) to address Pro/Con Iteration Plan for June 2019 #1
  2. Write documentation to address Approach Iteration Plan for June 2019 #1
  3. Add metadata to components and configuration yaml files to opt-in to hot reloading for all pods/applications to pick up this component instantly.
    • This will allow customers to rollout hot-patches or errors to applications without restarting entire application
  4. (optional) Add ability for applications/pod specs to be annotated/flagged to always pick up components instantly upon change, no matter what above metadata says

This will allow 3 cases:

Default Behavior

  1. Application running locally or kubernetes
  2. New configuration applied
  3. Application doesn't pick up new configuration until restarted/scaled

Apply Hot Reload to Component/Configuration

  1. Application running locally or kubernetes
  2. New configuration/component applied with metadata set to opt into hot reload
  3. All applications pick up new configuration/component

Apply Hot Reload to Application/Pod

  1. Application running locally or kubernetes with flag or annotation set to hot reload
  2. New configuration/component applied (with or without metadata for hot reload, doesn't matter)
  3. Application picks up new configuration/component

We could brand this as a new feature that users opt into if they want to.

@RadoslavGatev
Copy link

RadoslavGatev commented Sep 30, 2020

It seems the code for the hot reload is already in place. According to the operator logs, sidecars are successfully establishing connections to it and component changes are being successfully detected. However, for some reason sidecars are not getting notified about the modified component.

@yaron2 yaron2 added this to To do in 1.0.0 Milestone 1 via automation Sep 30, 2020
@yaron2
Copy link
Member

yaron2 commented Sep 30, 2020

@RadoslavGatev The operator side is mostly in place, however the sidecar part is currently (intentionally) ignoring updates.

We (maintainers) have spoken about this issue today and I just triaged this into the 0.12.0 milestone.

@msfussell msfussell added the size/M 2 weeks of work label Oct 7, 2020
@artursouza artursouza moved this from To do to In progress in 1.0.0-RC1 Milestone features Oct 8, 2020
@artursouza artursouza removed this from To do in 1.0.0 Milestone 1 Oct 15, 2020
@artursouza artursouza removed this from In progress in 1.0.0-RC1 Milestone features Oct 15, 2020
@artursouza artursouza added P2 and removed P1 labels Oct 15, 2020
@sebader
Copy link

sebader commented Feb 10, 2021

just came across the same issue. This really need to be solved. How else do you propose people can be sure config changes are applied?

@kamesh1677
Copy link

Any fix planned for this issue in the coming releases ?

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Issues and PRs without response label Jun 24, 2021
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 6, 2023
Branched from dapr#7239

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Kubernetes mode will watch for Component resource
updates from the Operator. If a resource is detected as changed (spec is
different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

---

When running in Kubernetes mode, the operator loader streams updates from
the Operator API. When an event occurs, the event type will trigger the
resource to be created, updated, or deleted. Updated components will be
closed, and then re-initialized.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 6, 2023
Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 6, 2023
Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 6, 2023
Branched from dapr#7239

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Kubernetes mode will watch for Component resource
updates from the Operator. If a resource is detected as changed (spec is
different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

---

When running in Kubernetes mode, the operator loader streams updates from
the Operator API. When an event occurs, the event type will trigger the
resource to be created, updated, or deleted. Updated components will be
closed, and then re-initialized.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 6, 2023
Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 11, 2023
Part of dapr#1172
Branched from dapr#7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 11, 2023
Part of dapr#1172
Branched from dapr#7260

Adds support for actor state store hot reloading.

PR updates the actors package so that the actor state store is dynamically
retrieved from the component store every time the state it invoked. The
passed static actor state store name previously used to retrieve the
state store has been removed.

The component store has been updated to track the current actor state
store. It is no longer tracked by the state processor, and instead the
state processor updates the latest in the component store.

Like before, the actor state store _must_ be configured on Dapr startup,
else the actor subsystem runtime will not be enabled, even if an actor
state store is hot reloaded later.

Adds actor hotreloading integration tests for the actor state store.

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr-docs that referenced this issue Dec 19, 2023
Adds documentation for Component Hot Reloading

Part of dapr/dapr#1172

Signed-off-by: joshvanl <me@joshvanl.dev>
yaron2 pushed a commit that referenced this issue Dec 22, 2023
* Hot Reloading: SelfHosted Componenthot reloading

Part of #1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates reconciler to wait for processor queue to be empty after
each event

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert processor pending resource queue to 0 channel

Signed-off-by: joshvanl <me@joshvanl.dev>

* Speed up hotreloading integration tests

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Move `tempDir` hot reload state test `TempDir()` to outer test scope to
prevent `RemoveAll` on busy sqlite .db files.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linitng

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix order of close and check bool in runtime_test.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates state features in integration tests

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 27, 2023
Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 27, 2023
Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 27, 2023
Branched from dapr#7239

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Kubernetes mode will watch for Component resource
updates from the Operator. If a resource is detected as changed (spec is
different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

---

When running in Kubernetes mode, the operator loader streams updates from
the Operator API. When an event occurs, the event type will trigger the
resource to be created, updated, or deleted. Updated components will be
closed, and then re-initialized.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Dec 27, 2023
Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>
yaron2 pushed a commit that referenced this issue Jan 8, 2024
* Hot Reloading: SelfHosted Componenthot reloading

Part of #1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates reconciler to wait for processor queue to be empty after
each event

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert processor pending resource queue to 0 channel

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linitng

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Hot Reloading: SelfHosted Componenthot reloading

Part of #1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Hot Reloading: Operator Componenthot reloading

Branched from #7239

Part of #1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Kubernetes mode will watch for Component resource
updates from the Operator. If a resource is detected as changed (spec is
different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

---

When running in Kubernetes mode, the operator loader streams updates from
the Operator API. When an event occurs, the event type will trigger the
resource to be created, updated, or deleted. Updated components will be
closed, and then re-initialized.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds hotreloading e2e config

Signed-off-by: joshvanl <me@joshvanl.dev>

* Use correct URL endpoints for hotreloading e2e save

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert integration test case timeout to 180 seconds

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix hotreloading_test.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* Use correct external URL target

Signed-off-by: joshvanl <me@joshvanl.dev>

* Lint code

Signed-off-by: joshvanl <me@joshvanl.dev>

* Hot Reloading: SelfHosted Componenthot reloading

Part of #1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linitng

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Set eventually in crypto test to 5 secs

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Jan 9, 2024
Part of dapr#1172
Branched from dapr#7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Jan 9, 2024
Part of dapr#1172
Branched from dapr#7260

Adds support for actor state store hot reloading.

PR updates the actors package so that the actor state store is dynamically
retrieved from the component store every time the state it invoked. The
passed static actor state store name previously used to retrieve the
state store has been removed.

The component store has been updated to track the current actor state
store. It is no longer tracked by the state processor, and instead the
state processor updates the latest in the component store.

Like before, the actor state store _must_ be configured on Dapr startup,
else the actor subsystem runtime will not be enabled, even if an actor
state store is hot reloaded later.

Adds actor hotreloading integration tests for the actor state store.

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Jan 14, 2024
Part of dapr#1172
Branched from dapr#7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Jan 18, 2024
Part of dapr#1172
Branched from dapr#7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>
JoshVanL added a commit to JoshVanL/dapr that referenced this issue Jan 25, 2024
Part of dapr#1172
Branched from dapr#7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>
artursouza pushed a commit that referenced this issue Jan 26, 2024
* Hot Reloading: HTTP Middleware Reloading

Part of #1172
Branched from #7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update pkg/middleware/http/http.go

Co-authored-by: Loong Dai <long0dai@foxmail.com>
Signed-off-by: Josh van Leeuwen <me@joshvanl.dev>

* Remove false comment

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update middleware tests due to new reserved dapr Component name

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix http channel test panic

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove workflow.dapr component from middleware integration test
metadata component checks

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: Josh van Leeuwen <me@joshvanl.dev>
Co-authored-by: Loong Dai <long0dai@foxmail.com>
@mukundansundar mukundansundar modified the milestones: v1.13, v1.14 Feb 8, 2024
@mukundansundar
Copy link
Contributor

@JoshVanL @yaron2 @artursouza
Moving this issue to 1.14 for tracking actor state after adding all related closed PRs to 1.13 for tracking.

@JoshVanL JoshVanL removed this from the v1.14 milestone Apr 16, 2024
cicoyle pushed a commit to cicoyle/dapr that referenced this issue May 24, 2024
* Hot Reloading: SelfHosted Componenthot reloading

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates reconciler to wait for processor queue to be empty after
each event

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert processor pending resource queue to 0 channel

Signed-off-by: joshvanl <me@joshvanl.dev>

* Speed up hotreloading integration tests

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Move `tempDir` hot reload state test `TempDir()` to outer test scope to
prevent `RemoveAll` on busy sqlite .db files.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linitng

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix order of close and check bool in runtime_test.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates state features in integration tests

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Cassandra Coyle <cassie@diagrid.io>
cicoyle pushed a commit to cicoyle/dapr that referenced this issue May 24, 2024
* Hot Reloading: SelfHosted Componenthot reloading

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates reconciler to wait for processor queue to be empty after
each event

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert processor pending resource queue to 0 channel

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linitng

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Hot Reloading: SelfHosted Componenthot reloading

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Hot Reloading: Operator Componenthot reloading

Branched from dapr#7239

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Kubernetes mode will watch for Component resource
updates from the Operator. If a resource is detected as changed (spec is
different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

---

When running in Kubernetes mode, the operator loader streams updates from
the Operator API. When an event occurs, the event type will trigger the
resource to be created, updated, or deleted. Updated components will be
closed, and then re-initialized.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds hotreloading e2e config

Signed-off-by: joshvanl <me@joshvanl.dev>

* Use correct URL endpoints for hotreloading e2e save

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert integration test case timeout to 180 seconds

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix hotreloading_test.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* Use correct external URL target

Signed-off-by: joshvanl <me@joshvanl.dev>

* Lint code

Signed-off-by: joshvanl <me@joshvanl.dev>

* Hot Reloading: SelfHosted Componenthot reloading

Part of dapr#1172

Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.

If enabled, Daprd in Selfhosted mode will watch for resource updates
from watching the resource directory(s). If a resource is detected as
changed (spec is different to what is currently loaded), then Dapr will
close/create/update the resource.

Supports Component _except_ HTTP middlware which requires further
changes to the HTTP server handler machinery. A warning is thrown if hot
reloading is enabled and a HTTP middleware is updated.

The hot reloader reconciler is made generic to enable other resource
types to be added in future.

---

When running in standalone mode, the disk loader watches the directories
passed by --resources-path for updates to yaml files containing
Components.

When an event occurs, the reconciler and differ are responsible for
determining whether any resources have been created, updated, or
deleted, by comparing the local store specs with that of the remote. If
any have changed, the resource is closed (if it exists), and then
re-initialized (if it exists). A resource will only be closed if it has
been deleted, only initialized if it has been created, and closed &
initialized if it has been updated.

We consider a resource to have changed generally if anything apart from
its Kubernetes metadata or type meta object meta has changed, and
therefore needs some kind of reloading.

---

Currently, if a reloading component errors and `spec.ignoreErrors=false`
then Daprd will gracefully close, like a component loaded at startup
today. It is intended that in future the component will be re-inited on
a backoff queue in future in the case of errors, even if
`spec.ignoreErros=true`.

HTTP middleware component hot reloading is not supported as it requires
further changes to the HTTP server handler machinery, and I didn't want
to grow this PR further.

To use the HotReloading feature, which is currently only available as an
alpha feature in Selfhosted mode, users need to add the following
Configuration to the target Daprd;

```yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: hotreloading
spec:
  features:
  - name: HotReload
    enabled: true
```

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds app ready checks for subscribed input bindings and pubsub

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linitng

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Linting

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Review comments

Signed-off-by: joshvanl <me@joshvanl.dev>

* Set eventually in crypto test to 5 secs

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Cassandra Coyle <cassie@diagrid.io>
cicoyle pushed a commit to cicoyle/dapr that referenced this issue May 24, 2024
* Hot Reloading: HTTP Middleware Reloading

Part of dapr#1172
Branched from dapr#7260

Adds support for HTTP middleware hot reloading.

PR adds support for HTTP middlewares to be dynamically updated. The HTTP
Middleware returns a middleware.HTTP pipleine built from a config spec.
This pipeline implements a single handler which it itself runs the spec
configured HTTP middleware chain. When a middleware Component is added
or removed, the HTTP middleware store is updated and both the HTTP
server and App HTTP channel's pipeline's chains are updated dynamically.

Like today, if a pipeline spec contains handlers whose names, version,
or type does not match that from the store, then the handler is skipped
for that chain. Pipeline handler order is preserved (i.e. reverse order).

The middleware store has been made generic so can be used in future for
planned gRPC middleware support.

Middleware init has been moved from the runtime channel manager to the
runtime processor init procedure to allow for dynamic loading. The
middleware HTTP manager is passed to the runtime processor to expose the
store, and relevant built pipelines passed to the HTTP app channel and
HTTP server.

Middleware integration tests has been added for daprd to check
functionality, ordering, and hotreloading,

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update pkg/middleware/http/http.go

Co-authored-by: Loong Dai <long0dai@foxmail.com>
Signed-off-by: Josh van Leeuwen <me@joshvanl.dev>

* Remove false comment

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update middleware tests due to new reserved dapr Component name

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix http channel test panic

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove workflow.dapr component from middleware integration test
metadata component checks

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: Josh van Leeuwen <me@joshvanl.dev>
Co-authored-by: Loong Dai <long0dai@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment