New container app revision fails to start but pulumi up does not fail

[bmbferreira]

What happened?

Currently if a new revision for the container app service fails to deploy (i.e. the container fails to boot) the service keeps the old revision running but the pulumi up command does not fail. This is an issue because on github actions the checks are green but the old version of the image is still running.

Example

1. You can use this example from the official examples repository: https://github.com/pulumi/examples/blob/master/azure-py-containerapps/__main__.py
2. Run pulumi up and deploy a new revision successfully.
3. Change the docker image so that it fails to boot, i.e. exits with code 1 when starting.
4. Run pulumi up. The command will be successful but on azure you can see that the new revision failed to start and the old revision is still running on azure.

Output of pulumi about

CLI          
Version      3.78.1
Go Version   go1.20.7
Go Compiler  gc

Plugins
NAME          VERSION
azure-native  2.6.0
cloudflare    5.10.0
docker        3.6.1
python        unknown
random        4.13.2

Host     
OS       darwin
Version  13.6
Arch     arm64

This project is written in python: executable='/opt/homebrew/bin/python3' version='3.11.5
'

Current Stack: organization/[CONFIDENTIAL]

TYPE                                                    URN
[CONFIDENTIAL]


Found no pending operations associated with [CONFIDENTIAL]

Backend        
Name           [CONFIDENTIAL]
URL           [CONFIDENTIAL]
User           [CONFIDENTIAL]
Organizations  

Dependencies:
NAME                 VERSION
fastapi              0.101.1
pip                  23.2.1
pulumi-azure-native  2.6.0
pulumi-cloudflare    5.10.0
pulumi-docker        3.6.1
pulumi-random        4.13.2
setuptools           68.2.2
wheel                0.41.2

Pulumi locates its logs in [CONFIDENTIAL]

Additional context

As a workaround I'm doing the following validation to check if the deployment was successful:

if pulumi.runtime.is_dry_run() == False:
    pulumi.Output.all(
        container_app.latest_ready_revision_name, container_app.latest_revision_name
    ).apply(
        # raise exception if the latest ready revision is not the latest revision
        lambda args: pulumi.info(
            f"Successfully deployed revision {args[0]}", container_app
        )
        if args[0] == args[1]
        else pulumi.error(
            f"Deployment of new revision failed. `latest_ready_revision_name` is different from `latest_revision_name`. `latest_ready_revision_name` is {args[0]} and `latest_revision_name` is {args[1]}.",
            container_app,
        )
    )

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mikhailshilkov]

Hi @bmbferreira thank you for raising this issue. We rely on Azure API status codes to decide whether an operation succeeds or not. If Azure reports an update as successful, there isn't really much we can do automatically to decide otherwise.

[bmbferreira]

Hi @mikhailshilkov. I understand, but the reality is that if the app container on azure ends up rolling back the deployment then the update was not successful after all 😞 IMO pulumi shouldn't be just a simple wrapper around azure API and would be a bit clever in detecting these things

[Arttii]

Is there no nice way to get the status of the newly deployed resource and fail in case it does not reach ready in a certain period of time? I tried doing this getContainerApp, but cannot figure it out yet.