Queue jobs in Orchestrator when no matching nodes are found

[wdbaruni]

When no enough nodes are available at the time the evaluation is processed, the Orchestrator will fail the job saying not enough nodes, which can be changed to just queue another evaluation with some delay (WaitUntil), and only fail after reaching user define timeout windows

Tasks

Beta Give feedback

1. ExecutionTimeout to fail executions instead of jobs #3974
   comp/scheduler +1
   
2. Decouple adding history events from state updates #3991
   comp/datastore comp/scheduler +2
   
3. No bid rejection for transient reasons. E.g. no available capacity #3992
   comp/scheduler +1
   
4. Atomic job store transactions #3993
   comp/datastore +1
   
5. Atomic enqueue of evaluations #3972
   comp/datastore comp/eval-broker type/enhancement +3
   
6. Backfill evaluations at startup #3999
   comp/eval-broker +1
   
7. Update evaluation state #4000
   comp/eval-broker +1
   
8. Add queue and total timeout values to jobs #4016
   th/specs +1
   
9. Housekeeping task to fail jobs exceeding total timeouts #4017
   comp/scheduler +1
   
10. Reschedule jobs if no nodes were found #4018
    comp/scheduler +1