-
|
It takes only a few seconds to run a job in the k8s cluster. It takes 10 seconds to arrange the same job in argo-workflow, and almost 10 seconds is spent on workflow scheduling. Is there any way to speed up the running of a job |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments
-
|
Are there many workflows in the cluster? You can adjust the qps and burst of workflow-controller. |
Beta Was this translation helpful? Give feedback.
-
|
just one workflows for test,when i test two task and use the dag,it will cost 20s but the pod just cost for 6s
…---Original---
From: "shuangkun ***@***.***>
Date: Sat, Mar 9, 2024 10:41 AM
To: ***@***.***>;
Cc: ***@***.******@***.***>;
Subject: Re: [argoproj/argo-workflows] how to speed up the job of theargo-workflow (Discussion #12766)
Are there many workflows in the cluster? You can adjust the qps and burst of workflow-controller.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I personally understand that most of the time is spent on workflow scheduling, and it takes about 10 seconds to complete each task until the next one starts. Is there any optimization that can speed up scheduling
剑指龙门
***@***.***
…------------------ 原始邮件 ------------------
发件人: "剑指龙门" ***@***.***>;
发送时间: 2024年3月9日(星期六) 中午11:23
***@***.******@***.***>;
***@***.***>;
主题: Re: [argoproj/argo-workflows] how to speed up the job of theargo-workflow (Discussion #12766)
just one workflows for test,when i test two task and use the dag,it will cost 20s but the pod just cost for 6s
---Original---
From: "shuangkun ***@***.***>
Date: Sat, Mar 9, 2024 10:41 AM
To: ***@***.***>;
Cc: ***@***.******@***.***>;
Subject: Re: [argoproj/argo-workflows] how to speed up the job of theargo-workflow (Discussion #12766)
Are there many workflows in the cluster? You can adjust the qps and burst of workflow-controller.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
If there are a large number of workflows in the cluster, slow scheduling is normal. If there is only a simple workflow in the cluster, scheduling is very fast. If there is a sample workflow, we can take a look at it together. |
Beta Was this translation helpful? Give feedback.
-
|
This sounds like a duplicate of #11275. The Controller environment variable You can also speed up Pod start-up time by caching images, having available resources (i.e. over-provisioning and not needing to auto-scale as much), and more. All of those are conventional k8s tuning knobs, not specific to Argo. #7144 also discusses the use-case of re-using Pods between tasks (which is not appropriate for all users). |
Beta Was this translation helpful? Give feedback.
This sounds like a duplicate of #11275. The Controller environment variable
DEFAULT_REQUEUE_TIMEcontrols the workflow queue rate limiter. You can decrease that, but that in turn can overwhelm your k8s API server. It currently defaults to10s.You can also speed up Pod start-up time by caching images, having available resources (i.e. over-provisioning and not needing to auto-scale as much), and more. All of those are conventional k8s tuning knobs, not specific to Argo.
#7144 also discusses the use-case of re-using Pods between tasks (which is not appropriate for all users).