Agent never reconnects after workspace update or reboot (Cloudinit VM)

[Toastyice]

I'm using a cloudinit template, after I Restart or update the workspace (Proxmox VM) the coder agent never reconnects.

I've found simular issues: #3485 #2970
(As far as I can tell the issue should be long resolved)

Does the token change on every update? Or is it supposed to refresh itself?
Thanks in advance

2023-07-08 10:32:39.541 [warn]  run exited with error ...
    error= run coordinator:
               github.com/coder/coder/agent.(*agent).run
                   /home/runner/actions-runner/_work/coder/coder/agent/agent.go:659
             - GET https://coder.<mydomain>/api/v2/workspaceagents/me/coordinate: unexpected status code 403: Agent trying to connect from non-latest build.
                Error: build is outdated

The cloud init config:

  user_data = <<EOT
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
hostname: ${lower(data.coder_workspace.me.name)}
users:
- name: ${lower(data.coder_workspace.me.owner)}
  sudo: ALL=(ALL) NOPASSWD:ALL
  shell: /bin/bash
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
export CODER_AGENT_TOKEN=${coder_agent.dev.token}
sudo --preserve-env=CODER_AGENT_TOKEN -u ${lower(data.coder_workspace.me.owner)} /bin/bash -c '${coder_agent.dev.init_script}'
--//--
EOT

coder version: v0.25.0
docker-image: ghcr.io/coder/coder
using external postgres
Cloudinit OS tried: Fedora 38, Almalinux 9, Ubuntu 22.04
template: https://github.com/bpmct/coder-templates/tree/main/proxmox-vm

[fredsilvi]

Same issue from my side.
I am under proxmox provider too (with a custom template).

[Toastyice]

Same issue from my side. I am under proxmox provider too (with a custom template).

My two cents
I was never able to get this working as a potential workaround you could try to use the bpg/proxmox terraform provider and create a VM that holds the userdisks for persistence and another VM that mounts this disk. Then when updating the template recreate the VM without touching the first one. https://registry.terraform.io/providers/bpg/proxmox/latest/docs/resources/virtual_environment_vm#example-attached-disks
I'll share my findings when I get to it.
Or mount an NFS share for persistence

[Toastyice]

@fredsilvi I got it working using the other proxmox terraform provider and the two vm approach. here is my quick and dirty template: https://github.com/Toastyice/coder-proxmox
I'll close this issue as it was not a coder problem.