Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request to consider adding ownerreferences to subtasks created by composed task as well #5767

Open
wqld opened this issue Apr 9, 2024 · 1 comment
Labels
area/composed-tasks Belongs to ctr area/task-scheduler Belongs to task scheduling for/composed-tasks For Composed Tasks
Milestone

Comments

@wqld
Copy link

wqld commented Apr 9, 2024

Problem description:

CronJob resources deployed to Kubernetes environments by Spring Cloud Dataflow have a successfulJobsHistoryLimit value of 3 by default.
By this setting, the maximum number of Task Pods created from a CronJob is three (Completed).

However, Sub Task Pods created from Composed Task may have problems with IP assignment, Max Pod count limit, etc. because they are in a situation where more than 3 Completed pods are maintained.

Solution description:

This is caused by the lack of an Ownerreferences setting.
Task Pods that are created from the flow CronJob -> Job -> Pod have Ownerreferences as normal. As a result, when the Job is deleted, the Task Pods are also cleaned up by the GC.

  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: Job
    name: test-statistics-daily-28543710
    uid: 8f2f58f3-7e68-48c6-bbc6-b9a5435117e5

However, Sub task pods that are additionally created by Composed Tasks are not automatically cleaned up because they do not have Ownerrefences.
This causes various issues and should be fixed so that Sub task pods are given the same Ownerreferences.

Additional context:

  • kubernetes: 1.28.6-eks
  • batch/v1 CronJob
  • spring-cloud-dataflow: 2.11.2
  • composed-task-runner: 2.11.2
@github-actions github-actions bot added the status/need-triage Team needs to triage and take a first look label Apr 9, 2024
@corneil corneil added type/bug Is a bug report area/composed-tasks Belongs to ctr area/task-scheduler Belongs to task scheduling for/composed-tasks For Composed Tasks and removed status/need-triage Team needs to triage and take a first look labels Apr 9, 2024
@wqld
Copy link
Author

wqld commented Apr 15, 2024

Until this issue is resolved, we want to manually clean up the completed sub pods, but we are struggling because we can't find any association between the composed task pod and the sub task pods that are created by the composed task pod.
Is there any way to distinguish which composed task pod a particular sub task pod is created from?

(Note that we have created two schedule resources from one task, running daily and monthly.
So there are two cronjobs, but the name format of the subtask pods that are created is the same.)

@cppwfs cppwfs removed the type/bug Is a bug report label May 29, 2024
@cppwfs cppwfs added this to the 3.0.x milestone May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/composed-tasks Belongs to ctr area/task-scheduler Belongs to task scheduling for/composed-tasks For Composed Tasks
Projects
None yet
Development

No branches or pull requests

3 participants