vmbackup cannot open a snapshot that it just created

[CaptSpify]

Describe the bug

When I run:
vmbackup -storageDataPath=/var/victoriametrics -snapshot.createURL=http://victoriametrics.somehost.com:8428/snapshot/create -dst=/mnt/backups/vmetrics

I get this error message:

2024-05-09T16:55:42.934Z        info    VictoriaMetrics/app/vmbackup/snapshot/snapshot.go:23    Creating snapshot
2024-05-09T16:55:43.211Z        info    VictoriaMetrics/app/vmbackup/snapshot/snapshot.go:47    Snapshot 20240509165542-17CDDE9977A3CE47 created
2024-05-09T16:55:43.211Z        fatal   VictoriaMetrics/app/vmbackup/main.go:73 cannot open snapshot at "/var/victoriametrics/snapshots/20240509165542-17CDDE9977A3CE47": open /var/victoriametrics/snapshots/20240509165542-17CDDE9977A3CE47: no such file or directory

But it did create the directory, so it does exist:

# ls -l /var/victoriametrics/snapshots/20240509165542-17CDDE9977A3CE47
total 8
drwxr-xr-x    2 root     root          4096 May  9 16:55 data
lrwxrwxrwx    1 root     root            55 May  9 16:55 indexdb -> ../../indexdb/snapshots/20240509165542-17CDDE9977A3CE47
drwxr-xr-x    2 root     root          4096 May  9 16:55 metadata```

### To Reproduce

`vmbackup -storageDataPath=/var/victoriametrics -snapshot.createURL=http://victoriametrics.somehost.com:8428/snapshot/create -dst=/mnt/backups/vmetrics`

### Version

The version does not show on this binary, despite the --help flag telling me that it should:

$ vmbackup --help | grep -i version
-version
Show VictoriaMetrics version
$ vmbackup -version

$

However, this is from the victoria-metrics_1.53.1+ds-1+b5_amd64 package

### Logs

_No response_

### Screenshots

_No response_

### Used command-line flags

_No response_

### Additional information

_No response_

[CaptSpify]

I just tried with the latest version, and I can't tell if it creates the snapshot directory or not, because it deletes it before it says that it cannot find it:

$ vmbackup-prod -storageDataPath=/var/victoriametrics -snapshot.createURL=http://victoriametrics.somehost.com:8428/snapshot/create -dst=fs:///mnt/backup/vmetrics
2024-05-09T17:22:20.123Z        info    VictoriaMetrics/lib/logger/flag.go:12   build version: vmbackup-20240425-145523-tags-v1.101.0-0-g5334f0c2c
2024-05-09T17:22:20.123Z        info    VictoriaMetrics/lib/logger/flag.go:13   command-line flags
2024-05-09T17:22:20.123Z        info    VictoriaMetrics/lib/logger/flag.go:20     -dst="fs:///mnt/backup/vmetrics"
2024-05-09T17:22:20.123Z        info    VictoriaMetrics/lib/logger/flag.go:20     -snapshot.createURL="secret"
2024-05-09T17:22:20.123Z        info    VictoriaMetrics/lib/logger/flag.go:20     -storageDataPath="/var/victoriametrics"
2024-05-09T17:22:20.124Z        info    VictoriaMetrics/app/vmbackup/main.go:67 Snapshot create url http://victoriametrics.somehost.com:8428/snapshot/create
2024-05-09T17:22:20.124Z        info    VictoriaMetrics/app/vmbackup/main.go:78 Snapshot delete url http://victoriametrics.somehost.com:8428/snapshot/delete
2024-05-09T17:22:20.124Z        info    VictoriaMetrics/lib/snapshot/snapshot.go:32     Creating snapshot
2024-05-09T17:22:20.520Z        info    VictoriaMetrics/lib/snapshot/snapshot.go:64     Snapshot 20240509172220-17CDDE9977A3CE4B created
2024-05-09T17:22:20.521Z        info    VictoriaMetrics/lib/snapshot/snapshot.go:75     Deleting snapshot 20240509172220-17CDDE9977A3CE4B
2024-05-09T17:22:20.521Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:119        starting server at http://127.0.0.1:8420/
2024-05-09T17:22:20.521Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:120        pprof handlers are exposed at http://127.0.0.1:8420/debug/pprof/
2024-05-09T17:22:20.555Z        info    VictoriaMetrics/lib/snapshot/snapshot.go:108    Snapshot 20240509172220-17CDDE9977A3CE4B deleted
2024-05-09T17:22:20.555Z        fatal   VictoriaMetrics/app/vmbackup/main.go:104        cannot create backup: cannot open snapshot at "/var/victoriametrics/snapshots/20240509172220-17CDDE9977A3CE4B": open /var/victoriametrics/snapshots/20240509172220-17CDDE9977A3CE4B: no such file or directory

$ vmbackup-prod -version
vmbackup-20240425-145523-tags-v1.101.0-0-g5334f0c2c

[zekker6]

@CaptSpify Could you please clarify whether -storageDataPath/var/victoriametrics is pointing to the same file system when running both VictoriaMetrics and vmbackup?

[CaptSpify]

Yup:

1 root      3h05 /victoria-metrics-prod -loggerTimezone=America/Los_Angeles -storageDataPath=/var/victoriametrics/

[zekker6]

@CaptSpify Are both vmbackup and victoria-metrics-prod running on the same machine?
I'm asking because -snapshot.createURL=http://victoriametrics.somehost.com:8428/snapshot/create flag value looks like a URL which is pointing to another machine. If vmbackup is running on a separate host that would explain why it cant open snapshot at the given path - because is stored on a machine where victoria-metrics-prod is running.

[CaptSpify]

Ah, that's it then. vmbackup won't run in the victoriametrics docker container, so I set it up on another host. If it only runs locally, then I'm confused why using a url to create the snapshot is needed?

[zekker6]

It is needed since it is possible to change a listening host and port by using -httpListenAddr and to enforce an authentication to create snapshot by using -snapshotAuthKey command-line flags at VictoriaMetrics single-node side.
If any of these options is used it is possible to change -snapshot.createURL to point to a correct address and use valid credentials.

[CaptSpify]

Ah gotcha. I just moved to an export/import model for backups instead of using vmbackup/vmrestore, so this is good to close. Thanks