"openreplay-ingress-nginx-admission" not found #1209

psociety · 2023-04-26T08:00:44Z

psociety
Apr 26, 2023

Provider: AWS
Guide: https://docs.openreplay.com/en/deployment/deploy-aws/

When i run the step openreplay -i DOMAIN_NAME i get:

client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" ValidatingWebhookConfiguration
client.go:339: [debug] validatingwebhookconfigurations.admissionregistration.k8s.io "openreplay-ingress-nginx-admission" not found
uninstall.go:148: [debug] purge requested for openreplay
Error: release openreplay failed, and has been uninstalled due to atomic being set: failed pre-install: timed out waiting for the condition

The machine specs are the ones from the guide (t3.large, 8GB RAM,etc.) as I used to have openreplay working, but the disk got filled and i had to reset the volume and start from scratch.
I tracked the RAM usage during the setup and it's never > 1.5GB, so it shouldn't be an issue.

If i try to install it without relaying on openreplay command, i get:

client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" ServiceAccount
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" ClusterRole
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" ClusterRoleBinding
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" Role
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" RoleBinding
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission-create" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job openreplay-ingress-nginx-admission-create with timeout of 5m0s
client.go:568: [debug] Add/Modify event for openreplay-ingress-nginx-admission-create: ADDED
client.go:607: [debug] openreplay-ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for openreplay-ingress-nginx-admission-create: MODIFIED
client.go:607: [debug] openreplay-ingress-nginx-admission-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
upgrade.go:434: [debug] warning: Upgrade "openreplay" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:84: [debug] pre-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.5.0/command.go:918
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571

Same happens if i alter /openreplay/scripts/hemlcharts/init.sh with:
helm upgrade --install openreplay ./openreplay -n app --create-namespace --wait -f ./vars.yaml --atomic --debug
to:
helm upgrade --install openreplay ./openreplay -n app --create-namespace -f ./vars.yaml --debug

if i add --wait 15m it stills fails after those 15min, so it's not a time issue.

rjshrjndrn · 2023-05-05T14:14:50Z

rjshrjndrn
May 5, 2023
Collaborator

can you please do openreplay -s after a failed installation.
If that's showing any failed pods
openreplay -l <pod-name-from-above-command>

2 replies

psociety May 5, 2023
Author

Are failed pods the ones that show as "Completed" and are not running? Because i see the nginx ingress there:

- Checking OpenReplay Components Status 
[INFO] OpenReplay Version 
/usr/bin/openreplay: line 147: /var/lib/openreplay/vars.yaml: No such file or directory
[INFO] Disk 
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        49G  5.7G   43G  12% /
[INFO] Memory 
              total        used        free      shared  buff/cache   available
Mem:          7.7Gi       1.2Gi       663Mi        13Mi       5.8Gi       6.2Gi
Swap:            0B          0B          0B
[INFO] CPU 
Linux 5.15.0-1019-aws #23~20.04.1-Ubuntu SMP Thu Aug 18 03:20:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
CPU Count: 2
[INFO] Kubernetes 
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.25.6+k3s1
[INFO] Openreplay Component 
NAME                                              READY   STATUS      RESTARTS   AGE
kyverno-9d48cbcf6-5w44m                           1/1     Running     0          9d
databases-migrate-wgs92                           0/2     Completed   0          9d
openreplay-ingress-nginx-admission-create-g8hnp   0/1     Completed   0          9d
NAME                     READY   STATUS    RESTARTS   AGE
redis-master-0           1/1     Running   0          9d
postgresql-0             1/1     Running   0          9d
minio-57d48668ff-2g29f   1/1     Running   0          9d

rjshrjndrn May 5, 2023
Collaborator

Hi Can you ping me in OR community slack? we can have a call to check what's going on.

harithay · 2023-08-01T22:55:27Z

harithay
Aug 1, 2023

Have a similar issue with following error output

client.go:310: [debug] Starting delete for "sourcemapreader-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "db-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "http-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "peers-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "ender-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx" ServiceAccount
client.go:310: [debug] Starting delete for "frontend-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "chalice-openreplay" ServiceAccount
client.go:310: [debug] Starting delete for "assist-openreplay" ServiceAccount
client.go:339: [debug] serviceaccounts "storage-openreplay" not found
client.go:339: [debug] serviceaccounts "integrations-openreplay" not found
client.go:339: [debug] serviceaccounts "ender-openreplay" not found
client.go:339: [debug] serviceaccounts "assets-openreplay" not found
client.go:339: [debug] serviceaccounts "db-openreplay" not found
client.go:339: [debug] serviceaccounts "heuristics-openreplay" not found
client.go:339: [debug] serviceaccounts "sink-openreplay" not found
client.go:339: [debug] serviceaccounts "alerts-openreplay" not found
client.go:339: [debug] serviceaccounts "peers-openreplay" not found
client.go:339: [debug] serviceaccounts "frontend-openreplay" not found
client.go:339: [debug] serviceaccounts "chalice-openreplay" not found
client.go:339: [debug] serviceaccounts "assist-openreplay" not found
client.go:339: [debug] serviceaccounts "openreplay-ingress-nginx" not found
client.go:339: [debug] serviceaccounts "sourcemapreader-openreplay" not found
client.go:339: [debug] serviceaccounts "http-openreplay" not found
client.go:310: [debug] Starting delete for "openreplay-ingress-nginx-admission" ValidatingWebhookConfiguration
client.go:339: [debug] validatingwebhookconfigurations.admissionregistration.k8s.io "openreplay-ingress-nginx-admission" not found
uninstall.go:148: [debug] purge requested for openreplay
Error: release openreplay failed, and has been uninstalled due to atomic being set: failed pre-install: job failed: BackoffLimitExceeded
helm.go:84: [debug] failed pre-install: job failed: BackoffLimitExceeded
release openreplay failed, and has been uninstalled due to atomic being set
helm.sh/helm/v3/pkg/action.(*Install).failRelease
        helm.sh/helm/v3/pkg/action/install.go:449
helm.sh/helm/v3/pkg/action.(*Install).reportToRun
        helm.sh/helm/v3/pkg/action/install.go:433
helm.sh/helm/v3/pkg/action.(*Install).performInstall
        helm.sh/helm/v3/pkg/action/install.go:361
runtime.goexit
        runtime/asm_amd64.s:1571

Output of the openreplay -s

root@ocean /var/lib/openreplay # openreplay -s
/usr/bin/openreplay: line 23: /var/lib/openreplay/vars.yaml: No such file or directory
[INFO] Using KUBECONFIG /etc/rancher/k3s/k3s.yaml 

- Checking OpenReplay Components Status 
[INFO] OpenReplay Version 
/usr/bin/openreplay: line 158: /var/lib/openreplay/vars.yaml: No such file or directory
[INFO] Disk 
Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        436G   67G  347G  17% /
[INFO] Memory 
               total        used        free      shared  buff/cache   available
Mem:            62Gi       5.1Gi        48Gi        12Mi       9.1Gi        56Gi
Swap:           79Gi          0B        79Gi
[INFO] CPU 
Linux ocean 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.2 LTS"
CPU Count: 20
[INFO] Kubernetes 
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.25.6+k3s1
[INFO] Openreplay Component 
NAME                                                       READY   STATUS       RESTARTS   AGE
kyverno-background-controller-5c666b7464-99ltz             1/1     Running      0          25m
kyverno-reports-controller-568d77df4d-6v99l                1/1     Running      0          25m
kyverno-cleanup-controller-655fcb44b8-gswqp                1/1     Running      0          25m
kyverno-admission-controller-5f5976bc68-nfcjh              1/1     Running      0          25m
databases-migrate-zz2w6                                    0/2     Init:Error   0          12m
kyverno-cleanup-admission-reports-28182170-bffkg           0/1     Completed    0          95s
kyverno-cleanup-cluster-admission-reports-28182170-fnxtk   0/1     Completed    0          95s
NAME                     READY   STATUS    RESTARTS   AGE
redis-master-0           1/1     Running   0          25m
postgresql-0             1/1     Running   0          25m
minio-64cb68cc9f-xwjl5   1/1     Running   0          12m

output of openreplay -l databases-migrate-zz2w6

root@ocean /var/lib/openreplay # openreplay -l databases-migrate-zz2w6
/usr/bin/openreplay: line 23: /var/lib/openreplay/vars.yaml: No such file or directory
[INFO] Using KUBECONFIG /etc/rancher/k3s/k3s.yaml 
+ app databases-migrate-zz2w6 › git
app databases-migrate-zz2w6 git + mkdir -p /opt/openreplay/openreplay
app databases-migrate-zz2w6 git + cd /opt/openreplay/openreplay
app databases-migrate-zz2w6 git + git clone https://github.com/openreplay/openreplay .
app databases-migrate-zz2w6 git Cloning into '.'...
app databases-migrate-zz2w6 git fatal: unable to access 'https://github.com/openreplay/openreplay/': Could not resolve host: github.com
app databases-migrate-zz2w6 git + ls /opt/openreplay/openreplay
app databases-migrate-zz2w6 git + git checkout v1.14.0
app databases-migrate-zz2w6 git fatal: not a git repository (or any parent up to mount point /opt)
app databases-migrate-zz2w6 git Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
app databases-migrate-zz2w6 git + exit 10
- app databases-migrate-zz2w6 › git

0 replies

friendofasquid · 2024-02-09T09:20:05Z

friendofasquid
Feb 9, 2024

Any resolution to this? I am facing similar issues.

0 replies

rjshrjndrn · 2024-02-09T09:25:04Z

rjshrjndrn
Feb 9, 2024
Collaborator

Means your nginx is not running. Can you please check `openreplay -s` Feb 9, 2024 10:20:15 AM Scott Arbeitman ***@***.***>:

…

Any resolution to this? I am facing similar issues. — Reply to this email directly, view it on GitHub[#1209 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AATR2OJVW6FKTZFO7Q2NC73YSXS47AVCNFSM6AAAAAAXMBWAK6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIMJWGY2TG]. You are receiving this because you commented. [Tracking image][https://github.com/notifications/beacon/AATR2OOTZGOPBAC6QW7FUBTYSXS47A5CNFSM6AAAAAAXMBWAK6WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAQBWY2.gif]

1 reply

friendofasquid Feb 12, 2024

Yup. For me, it was too small a hard-drive that probably prevented a clean start-up.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"openreplay-ingress-nginx-admission" not found #1209

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

"openreplay-ingress-nginx-admission" not found #1209

psociety Apr 26, 2023

Replies: 4 comments · 3 replies

rjshrjndrn May 5, 2023 Collaborator

psociety May 5, 2023 Author

rjshrjndrn May 5, 2023 Collaborator

harithay Aug 1, 2023

friendofasquid Feb 9, 2024

rjshrjndrn Feb 9, 2024 Collaborator

friendofasquid Feb 12, 2024

psociety
Apr 26, 2023

Replies: 4 comments 3 replies

rjshrjndrn
May 5, 2023
Collaborator

psociety May 5, 2023
Author

rjshrjndrn May 5, 2023
Collaborator

harithay
Aug 1, 2023

friendofasquid
Feb 9, 2024

rjshrjndrn
Feb 9, 2024
Collaborator