Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hot Reloading: Kubernetes Component hot reloading #7260

Merged
merged 33 commits into from
Jan 8, 2024

Commits on Dec 27, 2023

  1. Hot Reloading: SelfHosted Componenthot reloading

    Part of dapr#1172
    
    Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.
    
    If enabled, Daprd in Selfhosted mode will watch for resource updates
    from watching the resource directory(s). If a resource is detected as
    changed (spec is different to what is currently loaded), then Dapr will
    close/create/update the resource.
    
    Supports Component _except_ HTTP middlware which requires further
    changes to the HTTP server handler machinery. A warning is thrown if hot
    reloading is enabled and a HTTP middleware is updated.
    
    The hot reloader reconciler is made generic to enable other resource
    types to be added in future.
    
    ---
    
    When running in standalone mode, the disk loader watches the directories
    passed by --resources-path for updates to yaml files containing
    Components.
    
    When an event occurs, the reconciler and differ are responsible for
    determining whether any resources have been created, updated, or
    deleted, by comparing the local store specs with that of the remote. If
    any have changed, the resource is closed (if it exists), and then
    re-initialized (if it exists). A resource will only be closed if it has
    been deleted, only initialized if it has been created, and closed &
    initialized if it has been updated.
    
    We consider a resource to have changed generally if anything apart from
    its Kubernetes metadata or type meta object meta has changed, and
    therefore needs some kind of reloading.
    
    ---
    
    Currently, if a reloading component errors and `spec.ignoreErrors=false`
    then Daprd will gracefully close, like a component loaded at startup
    today. It is intended that in future the component will be re-inited on
    a backoff queue in future in the case of errors, even if
    `spec.ignoreErros=true`.
    
    HTTP middleware component hot reloading is not supported as it requires
    further changes to the HTTP server handler machinery, and I didn't want
    to grow this PR further.
    
    To use the HotReloading feature, which is currently only available as an
    alpha feature in Selfhosted mode, users need to add the following
    Configuration to the target Daprd;
    
    ```yaml
    apiVersion: dapr.io/v1alpha1
    kind: Configuration
    metadata:
      name: hotreloading
    spec:
      features:
      - name: HotReload
        enabled: true
    ```
    
    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    dcc4c02 View commit details
    Browse the repository at this point in the history
  2. Adds app ready checks for subscribed input bindings and pubsub

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    c807097 View commit details
    Browse the repository at this point in the history
  3. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    bb6b06d View commit details
    Browse the repository at this point in the history
  4. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    e824506 View commit details
    Browse the repository at this point in the history
  5. Updates reconciler to wait for processor queue to be empty after

    each event
    
    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    cc2243f View commit details
    Browse the repository at this point in the history
  6. Revert processor pending resource queue to 0 channel

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    c3a773c View commit details
    Browse the repository at this point in the history
  7. Review comments

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    b808b4d View commit details
    Browse the repository at this point in the history
  8. Linitng

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    c8efb48 View commit details
    Browse the repository at this point in the history
  9. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    2ce1d59 View commit details
    Browse the repository at this point in the history
  10. Adds review comments

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    81cd03b View commit details
    Browse the repository at this point in the history
  11. Hot Reloading: SelfHosted Componenthot reloading

    Part of dapr#1172
    
    Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.
    
    If enabled, Daprd in Selfhosted mode will watch for resource updates
    from watching the resource directory(s). If a resource is detected as
    changed (spec is different to what is currently loaded), then Dapr will
    close/create/update the resource.
    
    Supports Component _except_ HTTP middlware which requires further
    changes to the HTTP server handler machinery. A warning is thrown if hot
    reloading is enabled and a HTTP middleware is updated.
    
    The hot reloader reconciler is made generic to enable other resource
    types to be added in future.
    
    ---
    
    When running in standalone mode, the disk loader watches the directories
    passed by --resources-path for updates to yaml files containing
    Components.
    
    When an event occurs, the reconciler and differ are responsible for
    determining whether any resources have been created, updated, or
    deleted, by comparing the local store specs with that of the remote. If
    any have changed, the resource is closed (if it exists), and then
    re-initialized (if it exists). A resource will only be closed if it has
    been deleted, only initialized if it has been created, and closed &
    initialized if it has been updated.
    
    We consider a resource to have changed generally if anything apart from
    its Kubernetes metadata or type meta object meta has changed, and
    therefore needs some kind of reloading.
    
    ---
    
    Currently, if a reloading component errors and `spec.ignoreErrors=false`
    then Daprd will gracefully close, like a component loaded at startup
    today. It is intended that in future the component will be re-inited on
    a backoff queue in future in the case of errors, even if
    `spec.ignoreErros=true`.
    
    HTTP middleware component hot reloading is not supported as it requires
    further changes to the HTTP server handler machinery, and I didn't want
    to grow this PR further.
    
    To use the HotReloading feature, which is currently only available as an
    alpha feature in Selfhosted mode, users need to add the following
    Configuration to the target Daprd;
    
    ```yaml
    apiVersion: dapr.io/v1alpha1
    kind: Configuration
    metadata:
      name: hotreloading
    spec:
      features:
      - name: HotReload
        enabled: true
    ```
    
    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    eabeee6 View commit details
    Browse the repository at this point in the history
  12. Adds app ready checks for subscribed input bindings and pubsub

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    af2b64e View commit details
    Browse the repository at this point in the history
  13. Hot Reloading: Operator Componenthot reloading

    Branched from dapr#7239
    
    Part of dapr#1172
    
    Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.
    
    If enabled, Daprd in Kubernetes mode will watch for Component resource
    updates from the Operator. If a resource is detected as changed (spec is
    different to what is currently loaded), then Dapr will
    close/create/update the resource.
    
    Supports Component _except_ HTTP middlware which requires further
    changes to the HTTP server handler machinery. A warning is thrown if hot
    reloading is enabled and a HTTP middleware is updated.
    
    ---
    
    When running in Kubernetes mode, the operator loader streams updates from
    the Operator API. When an event occurs, the event type will trigger the
    resource to be created, updated, or deleted. Updated components will be
    closed, and then re-initialized.
    
    We consider a resource to have changed generally if anything apart from
    its Kubernetes metadata or type meta object meta has changed, and
    therefore needs some kind of reloading.
    
    ---
    
    Currently, if a reloading component errors and `spec.ignoreErrors=false`
    then Daprd will gracefully close, like a component loaded at startup
    today. It is intended that in future the component will be re-inited on
    a backoff queue in future in the case of errors, even if
    `spec.ignoreErros=true`.
    
    HTTP middleware component hot reloading is not supported as it requires
    further changes to the HTTP server handler machinery, and I didn't want
    to grow this PR further.
    
    To use the HotReloading feature, which is currently only available as an
    alpha feature in Selfhosted mode, users need to add the following
    Configuration to the target Daprd;
    
    ```yaml
    apiVersion: dapr.io/v1alpha1
    kind: Configuration
    metadata:
      name: hotreloading
    spec:
      features:
      - name: HotReload
        enabled: true
    ```
    
    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    29d3746 View commit details
    Browse the repository at this point in the history
  14. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    de78797 View commit details
    Browse the repository at this point in the history
  15. Adds hotreloading e2e config

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    f464bcf View commit details
    Browse the repository at this point in the history
  16. Use correct URL endpoints for hotreloading e2e save

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    be0b495 View commit details
    Browse the repository at this point in the history
  17. Revert integration test case timeout to 180 seconds

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    68e449b View commit details
    Browse the repository at this point in the history
  18. Fix hotreloading_test.go

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    56a6bdf View commit details
    Browse the repository at this point in the history
  19. Use correct external URL target

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    e65e961 View commit details
    Browse the repository at this point in the history
  20. Lint code

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    54a25b4 View commit details
    Browse the repository at this point in the history
  21. Hot Reloading: SelfHosted Componenthot reloading

    Part of dapr#1172
    
    Adds hot reloading functionality to Daprd, which is gated behind the HotReload preview feature.
    
    If enabled, Daprd in Selfhosted mode will watch for resource updates
    from watching the resource directory(s). If a resource is detected as
    changed (spec is different to what is currently loaded), then Dapr will
    close/create/update the resource.
    
    Supports Component _except_ HTTP middlware which requires further
    changes to the HTTP server handler machinery. A warning is thrown if hot
    reloading is enabled and a HTTP middleware is updated.
    
    The hot reloader reconciler is made generic to enable other resource
    types to be added in future.
    
    ---
    
    When running in standalone mode, the disk loader watches the directories
    passed by --resources-path for updates to yaml files containing
    Components.
    
    When an event occurs, the reconciler and differ are responsible for
    determining whether any resources have been created, updated, or
    deleted, by comparing the local store specs with that of the remote. If
    any have changed, the resource is closed (if it exists), and then
    re-initialized (if it exists). A resource will only be closed if it has
    been deleted, only initialized if it has been created, and closed &
    initialized if it has been updated.
    
    We consider a resource to have changed generally if anything apart from
    its Kubernetes metadata or type meta object meta has changed, and
    therefore needs some kind of reloading.
    
    ---
    
    Currently, if a reloading component errors and `spec.ignoreErrors=false`
    then Daprd will gracefully close, like a component loaded at startup
    today. It is intended that in future the component will be re-inited on
    a backoff queue in future in the case of errors, even if
    `spec.ignoreErros=true`.
    
    HTTP middleware component hot reloading is not supported as it requires
    further changes to the HTTP server handler machinery, and I didn't want
    to grow this PR further.
    
    To use the HotReloading feature, which is currently only available as an
    alpha feature in Selfhosted mode, users need to add the following
    Configuration to the target Daprd;
    
    ```yaml
    apiVersion: dapr.io/v1alpha1
    kind: Configuration
    metadata:
      name: hotreloading
    spec:
      features:
      - name: HotReload
        enabled: true
    ```
    
    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    d5d7be9 View commit details
    Browse the repository at this point in the history
  22. Adds app ready checks for subscribed input bindings and pubsub

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    d62e8d9 View commit details
    Browse the repository at this point in the history
  23. Linitng

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    f824261 View commit details
    Browse the repository at this point in the history
  24. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    ddc528b View commit details
    Browse the repository at this point in the history
  25. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    40fff23 View commit details
    Browse the repository at this point in the history
  26. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    4b2cbd8 View commit details
    Browse the repository at this point in the history
  27. Linting

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    19eb5ea View commit details
    Browse the repository at this point in the history
  28. Review comments

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Dec 27, 2023
    Configuration menu
    Copy the full SHA
    f3e5d60 View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2024

  1. Configuration menu
    Copy the full SHA
    e483e9e View commit details
    Browse the repository at this point in the history
  2. Review comments

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    83562fb View commit details
    Browse the repository at this point in the history
  3. Set eventually in crypto test to 5 secs

    Signed-off-by: joshvanl <me@joshvanl.dev>
    JoshVanL committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    f2d34c4 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Configuration menu
    Copy the full SHA
    49f7c97 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b76e6ac View commit details
    Browse the repository at this point in the history