Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lagoon PHP CLI images slow termination #976

Closed
smlx opened this issue Apr 18, 2024 · 1 comment · Fixed by #989
Closed

Lagoon PHP CLI images slow termination #976

smlx opened this issue Apr 18, 2024 · 1 comment · Fixed by #989

Comments

@smlx
Copy link
Member

smlx commented Apr 18, 2024

Lagoon php-8.x-cli images currently do not shut down gracefully in Kubernetes. Instead on deletion they ignore the graceful stop signal and are forcefully killed after 30 seconds. This means that it takes far longer than necessary to delete a cli pod, which has several impacts such as increasing Lagoon build rollout time.

The graceful shutdown signal is ignored because the Lagoon php-8.x-cli images are built on top of the upstream php-fpm image (via Lagoon's php-8.x-fpm). PHP's FPM uses SIGQUIT as a special signal to mean "graceful shutdown", so the upstream php-fpm image is configured with STOPSIGNAL SIGQUIT. Container runtimes generally respect the STOPSIGNAL metadata, so Kubernetes uses SIGQUIT too. Since we build the Lagoon php-8.x-cli on Lagoon's php-8.x-fpm, which is built on upstream php-fpm, the Lagoon php-8.x-cli image inherits STOPSIGNAL SIGQUIT. Unfortunately the docker-sleep script used in the CLI pod ignores the non-standard SIGQUIT.

You can see the scenario described above in the image metadata. Note that upstream php-cli doesn't specify a STOPSIGNAL, so falls back to the default SIGTERM:

screenshot_2024-04-18-215426

From what I can see there are a couple of options to fix this:

  1. Stop building Lagoon php-8.x-cli images on upstream php-fpm, and instead build them on upstream php-cli which uses the default STOPSIGNAL SIGTERM.
  2. Add STOPSIGNAL SIGTERM to Lagoon's php-8.x-cli Dockerfiles.

Here's the ~98% pod termination speedup observed when adding STOPSIGNAL SIGTERM to a Lagoon php-8.x-cli image (option 2):
screenshot_2024-04-18-213506

@rocketeerbkw
Copy link
Member

This also happens locally (with pygmy) when running docker-compose down, the cli always takes the longest to quit. I never considered it to be an image issue, so it would be great to fix it.

Stop building Lagoon php-8.x-cli images on upstream php-fpm, and instead build them on upstream php-cli which uses the default STOPSIGNAL SIGTERM.

What would be the advantage of that? The docs make it sound like there's nothing special that isn't included in the php-fpm image, and the php-fpm includes everything that php-cli does. Wouldn't we just be copy/pasting most of what's in our fpm Dockerfile anyway?

Add STOPSIGNAL SIGTERM to Lagoon's php-8.x-cli Dockerfiles.

That sounds like the easiest and best choice to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants