[4.6.0]: "A lock could not be acquired to modify the project config" during some of our deployments

[benediktblank]

What happened?

Description

For some time now, during deployments on our cluster, we get the error "A lock could not be acquired to modify the project config" when the deployment is in the process of applying the project config.

We apply the config with the "php craft project-config/apply" command. So only the changes/differences are applied.

This error fails the deployment. It restarts then, sometimes it goes through the second time, other times it fails again
Sometimes it takes 3 tries to get it done. Other times it goes through without errors the first time.
We've updated to 4.6.0, because I read something about Mutex in the change logs. But that did not change much in terms of our issue.

As far as I understand, it cannot acquire a mutex lock on the project config, when another process already holds a lock, right?
But I cannot say which process that would be, because the project config is only changed during the deployments.

We did not disable the "Settings" area in the backend, so one could still look into the config in the backend and technically also make changes. But it's not very likely that this is happening every time during a deployment. We don't make changes there at all, only via the project config files.

Are there any other places that acquire a lock on the project config that could cause the error during the deployment?

We use Redis for the Mutex Locks.

Steps to reproduce

I don't think it can be reproduced with a simple step by step manual.

Expected behavior

It goes through without the error.

Actual behavior

Sometimes the error happens.

Craft CMS version

4.6.0

PHP version

8.1.27

Operating system and version

Linux 5.15.0-1051-azure

Database type and version

MySQL 8.0.34

Image driver and version

Imagick 3.7.0 (ImageMagick 7.1.1-13)

Installed plugins and versions

Azure Blob Storage 2.0.0
Formie 2.1.3
Icon Picker dev-craft-4
Neo 3.9.11
Redactor 3.0.4
Retour 4.1.12
Sentry SDK 2.0.1
Super Table 3.0.12

[angrybrad]

@benediktblank guessing you're overriding the default mutext component in config/app.php to use Redis.

If so, remove that so it falls back to the default database mutex driver. I'm guessing that will solve your issue.