Image name not available when using watchtower or similar

[niawag]

Bug description
On standalone stacks (arm64 2.9.0) the image name is not available (only the sha256 fingerprint is) for auto-updating programs (such as watchtower), making the auto-updating process fail.

Expected behavior
The image name should be available, allowing the autoupdating process

Portainer Logs
//not relevant (I think)

Steps to reproduce the issue:

1. Create a stack containing at least one container
2. Run an auto-updating program such as Watchtower
3. Auto-updating fail because name image is not available

Technical details:

• Portainer version: 2.9.0
• Docker version (managed by Portainer): 19.03.15
• Platform (windows/linux): linux arm64
• Use Case (delete as appropriate): Using Portainer at Home
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context
This problem appeared with portainer 2.9.0 (on arm64 platforms) because https://github.com/docker/compose-cli is used for standalone stack. The bug has been reported here: containrrr/watchtower#1050 and is already solved here: docker-archive/compose-cli#1997
I'm filing this issue to let you know about it and to make sure the next compose-cli version will be merged to portainer to solve this.

Workaround
Use docker-compose (not docker compose, mind the -) from command line to deploy the stack instead of the portainer GUI.

[huib-portainer]

The use of compose-cli was introduced by #4776.

[niawag]

Exactly, and got released with portainer 2.9.0

[electron23]

Same here and with portainerci/portainer:develop (2.9.1) too.

Thanks for the workarround info.

[niawag]

I noticed I'm having the same problem on arm plateform (not arm64).

[marieldejesus12]

I'm having the same problem too. Same settings. In arm64 architecture too.

[BetaAthe]

I've the same issue in ARM64. Usually, the first deployment of the stack is correct, but updating the stack removes the image name.

[samdulam]

Confirmed on arm64, unable to reproduce on amd64

[niawag]

I edited my first post after noticing that I made a copy/paste mistake and haven't provided the upstream solution: docker-archive/compose-cli#1997
I think there is no problem on amd64 because it's not using https://github.com/docker/compose-cli

[huib-portainer]

So the current latest release of compose-cli is v1.0.17, and that merged PR isn't in any release yet.

[niawag]

You are unfortunately right! That's partly why I made this issue, as I said in my initial post:

Additional context This problem appeared with portainer 2.9.0 (on arm64 platforms) because https://github.com/docker/compose-cli is used for standalone stack. The bug has been reported here: containrrr/watchtower#1050 and is already solved here: docker/compose-cli#1997 I'm filing this issue to let you know about it and to make sure the next compose-cli version will be merged to portainer to solve this.

I hope a release with this PR is coming quickly!

[niawag]

I just checked and it is not entirely true, the PR was merged and released in v2.0.0-rc.2 available here: https://github.com/docker/compose-cli/releases/tag/v2.0.0-rc.2
But I'd totally understand if you don't want to merge a RC and wait for the stable release.

[niawag]

Hi, the corresponding PR (docker-archive/compose-cli#2038) is merged from v1.0.18 https://github.com/docker/compose-cli/releases/tag/v1.0.18 I suppose it's a smaller task to integrate than the v2RC.

[huib-portainer]

Sweet! Thanks for notifying us. We'll take a look at it shortly.

[SvenDowideit]

and now they're at v1.0.21 - https://github.com/docker/compose-cli/releases - and still "pre-release" :/

[N3m3515]

They have released 1.0.22 as pre-release 18 Days ago.

[huib-portainer]

You can give it a try by using the image portainerci/portainer:pr6225.
Please let us know how that's working for you.
Note that this is a development build and should not be used in a production environment.

[N3m3515]

I still get only image hashs instead of image names on redeployment of my images.
Can anyone else confirm?

[BetaAthe]

Can confirm that the problem persists, even when redeploying

[huib-portainer]

Thanks for the feedback!

[niawag]

Hi and thanks for the WIP, I've also tested it and am having the same results: hashes instead of names

Is it normal that the portainer version reported via the webui is 2.9.3 ?

[crystalcui8]

@Boergen I try to reproduce the issue you mentioned with below steps but can not reproduce it. Can you help to check what did I miss?

1. Create two stacks, one is with nginx and another is watchtower
2. Edit the stack of nginx via stack editor
3. Click 'Update the stack' to redeploy
4. The image name is correct after the redeployment successfully completed
   

[chevcheli0s]

I try to reproduce the issue you mentioned with below steps but can not reproduce it.

@crystalcui8 what is your architecture?

I have the same issue on 2.11.0. But it only happens on arm64. Even if I tag the containers manually, the name will not stick.
Two other instances with the same version on x86 are fine.

I've the same issue in ARM64. Usually, the first deployment of the stack is correct, but updating the stack removes the image name.

[crystalcui8]

@chevcheli0s I use ARM64 and the image portainerci/portainer:develop.
Note that this is a development build and should not be used in a production environment.

[Boergen]

@crystalcui8
I had watchtower already running, so i skipped that step... however, I do not think that it would matter.

• Created nginx stack --> Image name was correct
• Edited ngnix stack --> Image name was hashed again

And again to prove the workaround:

• Deleted the nginx stack but do NOT delete the image

• Add nginx stack again
  --> Image name still hashed

• Deleted the nginx stack

• Delete the image

• Add nginx stack again
  --> Image name correct

Same can be reproduced with all other stacks in my setup.

Running on a Raspberry Pi 4b with latest and updated 32bit Pi OS Lite (Buster).

[crystalcui8]

@Boergen I just reproduced the issue following your steps with 2.11.0, the Image name was hashed and the error message found in the log of watchtower as below.

But, the fix has been included in 2.11.1, so I repeated the steps with 2.11.1 and It works well.

[ahmedelakkad]

So currently it is mandatory to completely delete a stack and the image and add the stack again to keep the image name.

You can only delete the image and not the stack. And deploy existing stack. This will work until you redeploy stack with or without editing it.

Exactly!!

[RoggerFabri]

This issue still exists on 2.11.1.
Steps to reproduce:

1. Create stack helloworld1:

version: "2.4"

services:
  helloworld1:
    image: hello-world
    container_name: hello-world1

2. Create stack helloworld2:

version: "2.4"

services:
  helloworld2:
    image: hello-world
    container_name: hello-world2

First container is created correctly, second container has the hash instead of the name:

Tested on a raspberrypi4:

raspberrypi4
description: ARMv7 Processor rev 3 (v7l)
product: Raspberry Pi 4 Model B Rev 1.2
Raspbian GNU/Linux 10 (buster)

[crystalcui8]

@RoggerFabri I try to reproduce the issue you mentioned with 2.11.1 but failed. Can you please help to check what did I miss?

[RoggerFabri]

@RoggerFabri I try to reproduce the issue you mentioned with 2.11.1 but failed. Can you please help to check what did I miss?

Well... you did exactly as I did, your steps are correct. I tried it again, but I still have the issue mentioned above.
I'm running portainer from the following sha256: sha256:0e97a72a4154d69263ff89eb764587c377caac89a219041a0d3cfb9627f441e5 could you check if you're using the same?

[crystalcui8]

@RoggerFabri We use the different one.

[Boergen]

Updating to 2.11.1 also did not change the incorrect behaviour for me:

1. Had an exisiting stack of wireguard with a correct image name.
2. Edited and redeployed the stack
   --> hashed image name again

[chevcheli0s]

I have the same issue on my current installation.
I do a test with new config:

docker stop portainer
docker rm portainer
docker run -d -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /data/test/portainer:/data portainer/portainer-ce:latest

where /data/test/ is the new empty folder.
And with fresh install tests helloworld1 and helloworld2 successful passed

Than I compared the docker-compose files in the data directories(portainer/docker_config/cli-plugins) and the size was different.
And the solution is rm portainer/docker_config/cli-plugins/docker-compose and recreate container with portainer:

docker stop portainer
docker rm portainer
docker run -d -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /data/portainer:/data portainer/portainer-ce:latest

After that containers are created with correct image names.

[RoggerFabri]

I have the same issue on my current installation. I do a test with new config:

docker stop portainer
docker rm portainer
docker run -d -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /data/test/portainer:/data portainer/portainer-ce:latest

where /data/test/ is the new empty folder. And with fresh install tests helloworld1 and helloworld2 successful passed Than I compared the docker-compose files in the data directories(portainer/docker_config/cli-plugins) and the size was different. And the solution is rm portainer/docker_config/cli-plugins/docker-compose and recreate container with portainer:

docker stop portainer
docker rm portainer
docker run -d -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /data/portainer:/data portainer/portainer-ce:latest

After that containers are created with correct image names.

I can confirm that this solves the issue! Thank you, @chevcheli0s :)

[BetaAthe]

OK, I confirm that this solves the issue. Steps I followed:

1. Go to the portainer_data volume. Remove docker_config/cli-plugins/docker-compose
2. Run:

docker stop portainer
docker rm portainer
docker run -d -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data  portainer/portainer-ce

3. Redeploy all your stacks. You don't need to delete them, delete the images nor delete the containers.
4. Done.

[devedse]

@BetaAthe , @chevcheli0s, @RoggerFabri I tried this but it doesn't work for me:

I went to /var/lib/docker/volumes/portainer_data/_data/docker_config/cli-plugins, removed docker-compose there. Then recreated portainer as mentioned above, then checked if docker-compose was created again (which it was), then redeployed the stack.

Whenever I do docker inspect .... I still see the image name missing.

[chevcheli0s]

@devedse are you sure you are looking at the correct field?

Are you using the latest version?

[devedse]

@chevcheli0s , I did some more investigation and I found out that I was looking at the wrong field. The correct image name is now located in Config\Image.

So my issue has also been resolved, I was just checking the fix incorrectly :).

You can use this command to see all image names:

docker inspect $(docker ps -aq) | jq '.[].Config.Image'

[crystalcui8]

@RoggerFabri @BetaAthe @chevcheli0s I reproduced this issue under upgrade to 2.11.1 and the workaround works too. Thank you so much!

[crystalcui8]

The fix has been merged into develop branch. Can you please give it a try by using the image portainerci/portainer:develop?
Note that this is a development build and should not be used in a production environment.

[chaogeng77977]

@RoggerFabri @BetaAthe @chevcheli0s sorry to trouble you. Could you please give a try for the fix?

[NotSoFunnyClown]

Seems to be fixed with 2.11.1 :) Thank you <3

[ahmedelakkad]

Seems to be fixed with 2.11.1 :) Thank you <3

no same issue since 2.11.1 came out

[NotSoFunnyClown]

no same issue since 2.11.1 came out

Try to stop the stack, delete the image and then start the stack. It didn't work without deleting the image on my instance.

EDIT:
Okay there seems to be a bug anyway. Updating a stack results in a hash-image.

Reproduce:
When a stack is running and you update the docker-compose over the editor, the recreated container image has a hash again.
You have to stop the stack, delete the pulled image and start the stack again. So updating doesn't work :(

[crystalcui8]

@NotSoFunnyClown Can you share more ENV info? Which portainer image do you use? Is the issue under fresh install or upgrade?

[NotSoFunnyClown]

@NotSoFunnyClown Can you share more ENV info? Which portainer image do you use? Is the issue under fresh install or upgrade?

Yeah well... i see... I do not use the image where you fixed it :( My fault. Sorry.
Will it merged intoportainer/portainer-ce:latest?

[crystalcui8]

@NotSoFunnyClown It will merged into portainer/portainer-ce:latest later on but not soon. Can you please use the image portainerci/portainer:develop to have a try?
Note that this is a development build and should not be used in a production environment.

[NotSoFunnyClown]

@crystalcui8 Ok i now run portainerci/portainer:develop and it seems to fix the bug. I can't reproduce the bug anymore :)