Instance of CR in the work queue are not unique

[lmeunier-talend]

Hi,

I have a controller that reconcile resource of type A and it requeue instance of type A until the state on kubernetes match what i'm expecting

Now this controller also watch deployments which trigger my reconciling loop

My issue is that i'm watching like 5 of them and they often all change at the same time which mean i will queue 5 items of the same instance each time something happens.

Each item will be requeue until state match what is expected, then it will still proceed all remaining items in the queue even tho reconciliation is done for that instance

It's not that much of an issue but it does flood the logs for nothing and it happens that we have a http notification when reconciliation is done and i don't want to do 10 time in 1 sec

What I would like is to only queue if there are no instance of this object in work queue already so that i have never more than 1 instance of a object in the work queue

I did some search but i did not find a way to do that.

Thank you for answer

[alvaroaleman]

The work queue deduplicates identical objects, you will at most have a given identifier in there

[lmeunier-talend]

I have only 1 worker and yet according toe the log at the same second (2024-03-13T13:42:32Z) the operator does the loop 3 times for the same instance of my resources

10 second later which is my re-queue time, it does the same, 3 reconciliation loops

May be i'm missing but for me it means i have at least 3 instances of my object in the queue

[alvaroaleman]

Maybe you don't put a string but the full object into the workqueue, which then won't be de-duplicated because its not identical?

The mount of information you are giving here doesn't make it possible to help you, can you link to a code snippet that reproduces the behavior you are describing?

[troy0820]

/kind support

[lmeunier-talend]

my code only put instance in the queue in two ways:

• Owns(&appsv1.Deployment{})
• return ctrl.Result{RequeueAfter: r.RequeueAfter * time.Second}, nil

[lmeunier-talend]

The logs after i deleted two pods of watched deployment

2024-03-13T13:53:40Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2393b59b-3292-4ba3-a5d0-917ef42b720d"}
2024-03-13T13:53:40Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2393b59b-3292-4ba3-a5d0-917ef42b720d"}
2024-03-13T13:53:40Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "53e2a89f-c8a4-47ce-a30e-a4f0fe3d6e55"}
2024-03-13T13:53:40Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "53e2a89f-c8a4-47ce-a30e-a4f0fe3d6e55"}
2024-03-13T13:53:50Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "395670e8-ffae-49b0-9264-1abaee6541bb"}
2024-03-13T13:53:50Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "395670e8-ffae-49b0-9264-1abaee6541bb"}
2024-03-13T13:53:55Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "38fa5aa7-55a9-4886-908f-6373f9e5c1b6"}
2024-03-13T13:53:55Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "38fa5aa7-55a9-4886-908f-6373f9e5c1b6"}
2024-03-13T13:53:55Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "bdcec4e8-055c-4b4a-8cb9-7148a26e1c36"}
2024-03-13T13:53:55Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "bdcec4e8-055c-4b4a-8cb9-7148a26e1c36"}
2024-03-13T13:54:00Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "8cbf4a3d-753b-4fdc-9f79-bd2c149681ac"}
2024-03-13T13:54:00Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "8cbf4a3d-753b-4fdc-9f79-bd2c149681ac"}
2024-03-13T13:54:10Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "c427b9f9-209d-4d03-a0e2-0f756051aefe"}
2024-03-13T13:54:10Z	INFO	Install in progress, re-queueing	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "c427b9f9-209d-4d03-a0e2-0f756051aefe"}
2024-03-13T13:54:20Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "34cd6259-7ca7-44fc-b688-4152d3f86474"} found for service 'di-job-deployer'	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "34cd6259-7ca7-44fc-b688-4152d3f86474"}
2024-03-13T13:54:20Z	INFO	All services are running, exiting reconciliation loop	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "34cd6259-7ca7-44fc-b688-4152d3f86474"}
2024-03-13T13:54:20Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "7147644b-d077-4f96-8cd8-f2897ee81442"}
2024-03-13T13:54:20Z	INFO	All services are running, exiting reconciliation loop	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "7147644b-d077-4f96-8cd8-f2897ee81442"}
2024-03-13T13:54:20Z	INFO	Reconciling	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2dd9a973-bdcc-4867-b7d6-a1c516a59e09"}
2024-03-13T13:54:21Z	INFO	All services are running, exiting reconciliation loop	{"controller": "qlikruntime", "controllerGroup": "qlik.com", "controllerKind": "QlikRuntime", "QlikRuntime": {"name":"remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6"}, "namespace": "", "name": "remote-engine-dc4e5f42-0bab-46da-b43d-de24a13d25f6", "reconcileID": "2dd9a973-bdcc-4867-b7d6-a1c516a59e09"}

[alvaroaleman]

The logs after i deleted two pods of watched deployment

You likely are just getting multiple events from the Deployment, as its status is updated to reflect the two deletions in order and then new pods created by the depoyment controller. Additionally, while the queue itself de-duplicates, the queue might hold the object that is currently being reconciled, resulting in it getting reconciled again right after.