Imagerepositry: Flux is not able to scan new image tags from the artifactory

[SonalinP]

Flux is stopped scanning the image tags after 0.0.6 for one of image all of sudden .
i dont see any error in logs .
It just artifcatory has image tag as 0.0.7 and 0.0.8 as well . But flux is not able to scan post 0.0.6 .

flux: v2.1.0
helm-controller: v0.36.0
image-automation-controller: v0.36.0
image-reflector-controller: v0.30.0
kustomize-controller: v1.1.0
notification-controller: v1.1.0
source-controller: v1.1.0

ig ~/kubeconfig_pregl003
► annotating ImageRepository viaviservice in testapp-flux namespace
✔ ImageRepository annotated
◎ waiting for ImageRepository reconciliation
✔ scan fetched 6 tags

where as total number of tags available in image artifcatory is 8.

There is no change in manifest file/image path .
Why flux is suddenly stopped scanning latest images . Please guide further how to find out the root cause of this

[makkes]

Please post your complete ImageRepository manifest and the output of the following command:

flux logs --kind ImageRepository -A

[SonalinP]

We have installed flux pods in another ns not in flux-system . and ./flux logs are available for another ns other than flux-system.

kubectl get pods -n the-mission-flux 
NAME                                           READY   STATUS    RESTARTS   AGE
helm-controller-c7bfcddc4-cl68k                1/1     Running   7          11d
image-automation-controller-575c594f87-c7t7t   1/1     Running   6          9d
image-reflector-controller-65bdc6c986-mxjmz    1/1     Running   7          9d
kustomize-controller-76db95d867-w78ff          1/1     Running   8          11d
myapp                                          1/1     Running   2          195d
notification-controller-77554598f6-rfgl9       1/1     Running   4          11d
source-controller-6b54d9848c-q62wz             1/1     Running   6          11d

./flux logs --kind ImageRepository -n the-mission-flux  
✗ no Flux pods found in namespace "flux-system"

**ImageRepository manifest :**
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: ruservice
  namespace: testapp-flux
spec:
  image: <<artifactory path>>/ru-service
  interval: 1m0s

---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: viaviservice
  namespace: testapp-flux
spec:
  image: <artifactory path>/viavi-socket
  interval: 1m0s

---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: logservice
  namespace: testapp-flux
spec:
  image: <<artifactory path>/log-service
  interval: 1m0s

---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: teleservice
  namespace: testapp-flux
spec:
  image: <artifactory path>/telemetry-service
  interval: 1m0s

kubectl log of image-reflector-controller-65bdc6c986-mxjmz:

-42ad4a907fd1"}
{"level":"info","ts":"2023-09-15T15:47:17.425Z","msg":"no new tags found, next scan in 1m0s","controller":"imagerepository","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageRepository","ImageRepository":{"name":"ruservice","namespace":"testapp-flux"},"namespace":"testapp-flux","name":"ruservice","reconcileID":"ccf313f5-2af6-4a4a-a974-9cd1ad75a2cc"}
{"level":"info","ts":"2023-09-15T15:47:26.364Z","msg":"no new tags found, next scan in 1m0s","controller":"imagerepository","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageRepository","ImageRepository":{"name":"logservice","namespace":"testapp-flux"},"namespace":"testapp-flux","name":"logservice","reconcileID":"ea43e0ee-bdfc-467b-be01-09cf0786d1a6"}
{"level":"info","ts":"2023-09-15T15:47:28.524Z","msg":"no new tags found, next scan in 1m0s","controller":"imagerepository","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageRepository","ImageRepository":{"name":"viaviservice","namespace":"testapp-flux"},"namespace":"testapp-flux","name":"viaviservice","reconcileID":"b6f268b8-005b-46de-bcd2-5e99f7cf4ecb"}

but there is new tags available 0.0.7 and 0.0.8 in artifactory .

from ./flux get all -n testapp-flux
imagerepository/viaviservice    2023-09-15T17:50:32+02:00       False           True    successful scan: found 6 tags

imagepolicy/viaviservice        <artifcatory path>/viavi-socket:0.0.6 True    Latest image tag for '<artifactory path>/viavi-socket' resolved to 0.0.6

[makkes]

can you do a crane ls <artifcatory path>/viavi-socket so we can actually see the tags? Maybe they're not exposed right or anything like that. (details on how to install crane here).

[SonalinP]

./crane ls /viavi-socket
0.0.1
0.0.2
0.0.3
0.0.4
0.0.5
0.0.6
..
it is not listing 0.0.7 and 0.0.8

[makkes]

Then the tags don't exist. :-) Flux can't find what's not there.

[SonalinP]

The tags are available in artifactory . but it is neither able detect by flux nor by crane tool .

[makkes]

Can you pull them with docker pull <artifcatory path>/viavi-socket:0.0.8?

[SonalinP]

yes I am able to pull via docker pull

[makkes]

The call to "https:///v2//viavi-socket/tags/list?n=1000" only returns the tags [ "0.0.1", "0.0.2", "0.0.3", "0.0.4", "0.0.5", "0.0.6" ]. This is an error on Artifactory's side and nothing Flux (or any other tool discovering tags) can do anything about.

[hiddeco]

Given that it appears to be unable to see any images published after 0.0.6, is there a chance you e.g. changed the permissions some time after this release and forgot to include list permissions? That's one way I can think (without knowing anything about Artifactory's permission system) it would be able to pull the image, but wouldn't show up in the tag list.

[johnlabarge]

Did you ever figure this out? I think I'm seeing this as well.

[johnlabarge]

Also crane is showing all my tags so that's not the issue here. Image repository shows a random set of 4 that are by far not the latest.

[johnlabarge]

I think I figured out part of it I have a SNAPSHOT extension e.g. 0.0.1-SNAPSHOT-1 and when there is another digit after the first one after snapshot (e.g. 19) it ignores the tag altogether. :/