Feature description

Syntax:

id: check_for_job_status
namespace: dev

tasks:
  - id: databricks_job
    type: io.kestra.core.tasks.flows.WaitFor
    condition: "{{ (outputs.poll.body | jq('.state.result_state') | first) == 'SUCCESS' }}"
    task:
      id: poll
      type: io.kestra.plugin.fs.http.Request
      uri: https://reqres.in/api/job-status
      method: GET
      contentType: application/json
    checkFrequency:
      interval: PT1S # type: Duration
      maxDuration: PT1H # type: Duration
      maxAttempts: 50 # type: int

Icon:

The reason for this task (and the implementation):

1. Implementing a WhileLoop that would satisfy multiple use cases in one go proved challenging as we cannot promise this level of statefulness
2. Implementing a flowable task that could execute multiple tasks within the loop could break the executor
3. We don't implement all the same functionality as we do on the retry in the checkFrequency because this task is meant for waiting until the condition is satisfied (blocking calls) rather than for fetching data from APIs, which might require exponential retries.

For those reasons, we decided to implement this as a runnable task that can execute a single task. Later, we may explore the WhileLoop once we figure out a solution to the second issue as part of the ForEachItem redesign.