schedule pending workflows in reverse order

[constantin-ungureanu-github]

I have a very specific use case, that periodically triggers lot of workflows to be executed (through a cron workflow). All those new workflows will be queued and executed when a semaphore is available. All is good, things are running as expected.

However, I want to have control over what will be the next workflow to be executed. More specifically, I want the newest ones to run first, aka LIFO. By default this is FIFO.

I know there is a priority flag, however, this is not helpful in this case because I keep triggering new workflows, so even if I increase the priority, then the next wave should have higher priority and so on.

I also know there is a filtering mechanism for the pending workflows to be executed, but the only filtering I see is on time intervals, but noting for comparing pending workflows between themselves.

How would it be possible to have this functionality?

[shuangkun]

Another scheduling strategy may be needed.

[constantin-ungureanu-github]

is this functionality of selecting schedule strategy available? if it is, how ?

[constantin-ungureanu-github]

if you read my initial description, I explained why priority is not an option for my use case. basically every new wave of workflows would have to have a higher priority, but there is a limit, and then what ? the question is how can it be implemented the lifo instead of fifo for queueing workflows of SAME PRIORITY? this is an idea, is not a bug, I know is not available now, I tried various options. I can even try to implement this myself, but I need guidance. Kind Regards, Constantin Ungureanu

…

On Tue, Mar 26, 2024, 20:00 Anton Gilgur ***@***.***> wrote: No, it does not exist. You can set a priority <https://argo-workflows.readthedocs.io/en/latest/fields/#workflowspec> for Workflows to work around that manually though. — Reply to this email directly, view it on GitHub <#12648 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7K56KEKKYFBH4Q4VSMA63Y2HATLAVCNFSM6AAAAABDBHUCYCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSMJZGM4TQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[agilgur5]

if you read my initial description, I explained why priority is not an option for my use case.

Yea sorry, my bad. I deleted my comment almost immediately after posting it because of that. Looks like you still got an email notification though.
I read it a few times and that evaded me at least two of those readings. I changed the formatting a bit so it's more distinct.

basically every new wave of workflows would have to have a higher priority, but there is a limit, and then what ?

There's probably a way to workaround this, but I can't remember off the top of my head if there's a monotonically increasing algorithm that handles overflow without things like BigInts.

is this functionality of selecting schedule strategy available? if it is, how ?

I know is not available now

You asked before and no one explicitly answered, so just wanted to make the clear!
But yea I noticed you marked it as an "Idea" and it's in Discussions.

I can even try to implement this myself, but I need guidance.

It is processed as a queue, using k8s/client-go's workqueue under-the-hood. Changing the underlying data structure may entail a very large refactoring. And a "LIFO queue" is more commonly known as a "stack", so renames may make that even more complex. If you can do a user-land workaround, it would probably be easier, tbh.

[constantin-ungureanu-github]

Thanks for the answer! I am not familiar with internals of argo workflows, so your reply is of good help for me. How about the possibility to select the next item to be run based on some filtering/aggregation? Say the item with most recent timestamp to be selected. How would this be possible? In essence it should be similar with selection based on priority, but on a different field.

…

On Thu, Mar 28, 2024, 23:36 Anton Gilgur ***@***.***> wrote: if you read my initial description, I explained why priority is not an option for my use case. Yea sorry, my bad. I deleted my comment almost immediately after posting it because of that. Looks like you still got an email notification though. I read it a few times and that evaded me at least two of those readings. I changed the formatting a bit so it's more distinct. basically every new wave of workflows would have to have a higher priority, but there is a limit, and then what ? There's probably a way to workaround this, but I can't remember off the top of my head if there's a monotonically increasing algorithm that handles overflow without things like BigInts. is this functionality of selecting schedule strategy available? if it is, how ? I know is not available now You asked before <#12648 (reply in thread)> and no one explicitly answered, so just wanted to make the clear! But yea I noticed you marked it as an "Idea" and it's in Discussions. I can even try to implement this myself, but I need guidance. It is processed as a queue <https://github.com/argoproj/argo-workflows/blob/748ae475d3400f2aa51800c2337607811550d989/workflow/controller/controller.go#L127>, using k8s/client-go's workqueue <https://pkg.go.dev/k8s.io/client-go/util/workqueue> under-the-hood. Changing the underlying data structure may entail a *very* large refactoring. And a "LIFO queue" is more commonly known as a "stack", so renames may make that even more complex. If you can do a user-land workaround, it would probably be easier, tbh. — Reply to this email directly, view it on GitHub <#12648 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7K56I3BT2NXZIID4B33JLY2SLNLAVCNFSM6AAAAABDBHUCYCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBWGI3TQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>