Clone GitHub repos to the local Gitea during install #212

kispaljr · 2023-11-03T15:49:21Z

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

It clones GitHub repositories that are registered in porch to the local Gitea during install.
This is needed to make the sandbox more self-contained and it is also a first step toward making it work behind a corporate http proxy (because it avoids Internet traffic originating in the clusters).

Which issue(s) this PR fixes:

It is a step toward fixing nephio-project/nephio#409 , but it will not completely fix the issue.

** Notes to reviewer:
Git cloning is deliberately done directly on the host machine instead of a pod running in a management cluster. This way we only need to assume that the localhost can reach the Internet (i.e. http proxy vars are setup correctly), but we don't have to assume that workloads running in the clusters can also reach the internet (that would require the errorprone setting of the no_proxy variable, that is almost impossible to do correctly.)

Does this PR introduce a user-facing change?:

no.

…ernet traffic originating in the clusters later.

nephio-prow · 2023-11-03T15:49:32Z

Hi @kispaljr. Thanks for your PR.

I'm waiting for a nephio-project member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…ned since my original solution

kispaljr · 2023-11-03T16:21:46Z

/assign @electrocucaracha

@electrocucaracha, I see that you also made changes on how we handle stock git repos. Could you have a look if this was in line with your thinking?

electrocucaracha

I'm not able to test it, but I made few suggestions on it. Don't forget to include the license headers on the files

e2e/provision/playbooks/mirror_git_repo_to_gitea.yaml

e2e/provision/playbooks/cluster.yml

electrocucaracha · 2023-11-03T16:35:22Z

/assign @electrocucaracha

@electrocucaracha, I see that you also made changes on how we handle stock git repos. Could you have a look if this was in line with your thinking?

I put some comments on the draft. I think that makes sense, my only concern now it that prow hasn't triggered the CI yet.

kispaljr · 2023-11-03T21:22:53Z

/assign @electrocucaracha
@electrocucaracha, I see that you also made changes on how we handle stock git repos. Could you have a look if this was in line with your thinking?

I put some comments on the draft. I think that makes sense, my only concern now it that prow hasn't triggered the CI yet.

I think somebody (maybe @radoslawc ?) has to add an ok-to-test label to enable the CI tests

electrocucaracha · 2023-11-03T21:25:46Z

/ok-to-test

nephio-prow · 2023-11-03T21:26:58Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from electrocucaracha by writing /assign @electrocucaracha in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kispaljr · 2023-11-03T21:29:39Z

/retest

kispaljr · 2023-11-03T21:57:14Z

Can someone help me with the install-integration and provision-linter tests? Is it an error in my code or in the tests?

radoslawc · 2023-11-03T22:24:47Z

Can someone help me with the install-integration and provision-linter tests? Is it an error in my code or in the tests?

@kispaljr that's a problem with test images, I've pushed the fix here #213

kispaljr · 2023-11-03T23:33:14Z

/kind cleanup

electrocucaracha · 2023-11-09T16:00:46Z

e2e/provision/playbooks/roles/install/molecule/default/tasks/deploy_kpt.yml

+- name: "Fetch kpt package: {{ item }}"
+  ansible.builtin.command:
+    cmd: "kpt pkg get --for-deployment --context {{ k8s.context }} '{{ nephio_example_repo_uri }}/{{ item }}@{{ nephio_pkg_version }}'"
+    chdir: "/tmp"
+  changed_when: false
+
+- name: "Render kpt package: {{ item }}"
+  ansible.builtin.command:
+    cmd: "kpt fn render --context {{ k8s.context }}"
+    chdir: "/tmp/{{ item }}"
+  changed_when: false
+
+- name: "Init kpt package: {{ item }}"
+  ansible.builtin.command:
+    cmd: "kpt live init --namespace default --context {{ k8s.context }}"
+    chdir: "/tmp/{{ item }}"
+  changed_when: false
+
+- name: "Apply kpt package: {{ item }}"
+  ansible.builtin.command:
+    cmd: "kpt live apply --namespace default --reconcile-timeout 3600 --context {{ k8s.context }}"
+    chdir: "/tmp/{{ item }}"
+  changed_when: false


There is a kpt Ansible module that you can use if you need and also a kpt Ansible role as well.

kispaljr · 2023-11-10T12:35:36Z

/retest

kispaljr · 2023-11-10T13:50:50Z

/retest

kispaljr · 2023-11-10T14:09:01Z

@electrocucaracha could you take a look?

the install role now works as follows:
- expects a local_git input parameter with the fetch and push URIs of the Nephio management cluster-local git repo (it is Gitea in our case)
- it is theoretically Gitea-independent, but right now our Repository resource only supports creating a new repo in Gitea
- it uses only std git operations to copy external (Github) stock repos to the local git server
- the molecule test of the role is now deploys gitea and metallb before the actual test
kpt CLI calls were replaced with the kpt Ansible module, as you suggested
all linter findings (including the ones not in my code) were fixed, or was marked with a noqa tag, as you sugested

kispaljr · 2023-11-10T14:15:56Z

/retest

…stem namespace

…io-system namespace

kispaljr · 2023-11-10T15:21:54Z

/retest

…prepare playbook to converge playbook

kispaljr · 2023-11-12T13:36:46Z

/retest

electrocucaracha · 2023-11-14T01:21:19Z

@kispaljr I started testing this PR and putting some changes on my fork, I'll do a PR on your branch once it's ready

kispaljr · 2023-11-16T12:35:43Z

@kispaljr I started testing this PR and putting some changes on my fork, I'll do a PR on your branch once it's ready

Thank you. I am struggling to figure out why the webui ReplicaSet times out consistently in the install-integration test.

electrocucaracha · 2023-11-16T17:01:33Z

@kispaljr I started testing this PR and putting some changes on my fork, I'll do a PR on your branch once it's ready

Thank you. I am struggling to figure out why the webui ReplicaSet times out consistently in the install-integration test.

I've submitted a PR in your fork to apply most of the suggestions. It's hard to review given that your fork requires rebase some commits, but if you want to take a look these are the changes

kispaljr · 2023-11-20T14:55:04Z

@kispaljr I started testing this PR and putting some changes on my fork, I'll do a PR on your branch once it's ready

Thank you. I am struggling to figure out why the webui ReplicaSet times out consistently in the install-integration test.

I've submitted a PR in your fork to apply most of the suggestions. It's hard to review given that your fork requires rebase some commits, but if you want to take a look these are the changes

@electrocucaracha
I rebased your changes and pushed to the PR as-is. I have two comments:

unfortunately it didn't solve my problem with the install-integration test, it is still failing consistently with a webui deployment timeout. do you have an idea how to solve that?
removing all gitea specific input parameters from the install role and replacing it with generic "local git server" parameters was intentional from my part. I'd like to see the install role to be as independent from the actual kind of the git server that we deploy into the management cluster. I see that you reverted those changes. What is the reason behind this?

electrocucaracha · 2023-11-27T17:53:37Z

@kispaljr I'm still seeing some differences between these changes and this other

kispaljr · 2023-12-01T17:04:18Z

/test install-integration

kispaljr · 2023-12-03T18:04:42Z

``> @kispaljr I'm still seeing some differences between these changes and this other

@electrocucaracha , you are right, they were different. So, I made the tip of my PR (0123635) identical to the tip of your PR (5a5c342).
Unfortunately the install-integration test consistently fails the exact same way as before. Do you have any idea why? Should I just adjust a timeout somewhere in the webui deployment process?

Here is the status of the nephio-webui ReplicaSet in the install-integration test logs for reference:

"status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2023-12-01T17:14:00Z",
                        "lastUpdateTime": "2023-12-01T17:14:00Z",
                        "message": "Deployment does not have minimum availability.",
                        "reason": "MinimumReplicasUnavailable",
                        "status": "False",
                        "type": "Available"
                    },
                    {
                        "lastTransitionTime": "2023-12-01T17:24:01Z",
                        "lastUpdateTime": "2023-12-01T17:24:01Z",
                        "message": "ReplicaSet \"nephio-webui-5464c8dc64\" has timed out progressing.",
                        "reason": "ProgressDeadlineExceeded",
                        "status": "False",
                        "type": "Progressing"
                    }
                ],
                "observedGeneration": 1,
                "replicas": 1,
                "unavailableReplicas": 1,
                "updatedReplicas": 1

electrocucaracha · 2023-12-04T18:37:50Z

I tried to replicate it locally, but it worked. I'll give a second try to see if something changes.

Regarding the e2e, it seems like it had issues to bring the Tunnel bridge up

/retest

nephio-prow · 2023-12-04T19:52:16Z

@kispaljr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
install-integration	`0123635`	link	true	`/test install-integration`
e2e-ubuntu-focal	`0123635`	link	true	`/test e2e-ubuntu-focal`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

electrocucaracha · 2023-12-06T17:58:02Z

``> @kispaljr I'm still seeing some differences between these changes and this other

@electrocucaracha , you are right, they were different. So, I made the tip of my PR (0123635) identical to the tip of your PR (5a5c342). Unfortunately the install-integration test consistently fails the exact same way as before. Do you have any idea why? Should I just adjust a timeout somewhere in the webui deployment process?

Here is the status of the nephio-webui ReplicaSet in the install-integration test logs for reference:
"status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2023-12-01T17:14:00Z",
                        "lastUpdateTime": "2023-12-01T17:14:00Z",
                        "message": "Deployment does not have minimum availability.",
                        "reason": "MinimumReplicasUnavailable",
                        "status": "False",
                        "type": "Available"
                    },
                    {
                        "lastTransitionTime": "2023-12-01T17:24:01Z",
                        "lastUpdateTime": "2023-12-01T17:24:01Z",
                        "message": "ReplicaSet \"nephio-webui-5464c8dc64\" has timed out progressing.",
                        "reason": "ProgressDeadlineExceeded",
                        "status": "False",
                        "type": "Progressing"
                    }
                ],
                "observedGeneration": 1,
                "replicas": 1,
                "unavailableReplicas": 1,
                "updatedReplicas": 1

There is a way to run integration tests locally and review the state of the VM, you can use the following instruction tox -e install -- --destroy never on the e2e/provision

Clone GitHub to the local Gitea during install, in order to avoid Int…

d20ba2e

…ernet traffic originating in the clusters later.

nephio-prow bot added the kind/bug label Nov 3, 2023

nephio-prow bot requested review from johnbelamaric and radoslawc November 3, 2023 15:49

nephio-prow bot added the needs-ok-to-test label Nov 3, 2023

Fix some whitespaces

7d0addd

kispaljr marked this pull request as draft November 3, 2023 16:02

nephio-prow bot added the do-not-merge/work-in-progress label Nov 3, 2023

Adapting to Victor's changes regarding stock repo handling that happe…

f304818

…ned since my original solution

kispaljr marked this pull request as ready for review November 3, 2023 16:20

nephio-prow bot removed the do-not-merge/work-in-progress label Nov 3, 2023

nephio-prow bot assigned electrocucaracha Nov 3, 2023

electrocucaracha reviewed Nov 3, 2023

View reviewed changes

nephio-prow bot added ok-to-test and removed needs-ok-to-test labels Nov 3, 2023

kispaljr added 2 commits November 3, 2023 22:26

Incorporate changes suggested by Victor

66f93cc

Add missing copyright header

0b4606c

Making ansible-lint a bit happier

46b398d

Decrease wait time for creating the new repo in gitea

046db7e

radoslawc mentioned this pull request Nov 3, 2023

Fixing path for images that do not have git installed #213

Merged

electrocucaracha reviewed Nov 9, 2023

View reviewed changes

kispaljr added 2 commits November 10, 2023 12:11

Fix install/molecule test

d3063ef

Fix install/molecule test

85b72b9

kispaljr added 3 commits November 10, 2023 13:49

Replace calling kpt CLI with kpt ansible module

c810d3b

ansible-lint

73bfec2

Fix install/molecule test

62694fc

kispaljr added 2 commits November 10, 2023 15:22

Restore creating the Gitea username/password secret also in nephio-sy…

9473f78

…stem namespace

Fix: Restore creating the Gitea username/password secret also in neph…

73ca0b4

…io-system namespace

kispaljr added 3 commits November 10, 2023 16:57

debugging install role

747a8a4

debugging install role

9a1564e

Fix molecule test of install role: move setting local git facts from …

4ca8526

…prepare playbook to converge playbook

kispaljr added 2 commits December 1, 2023 17:12

Merge nephio/main into github-mirroring to be on the same base as Victor

5c0461e

Victor's changes

0123635

kispaljr force-pushed the github-mirroring branch from 0d5f139 to 0123635 Compare December 1, 2023 16:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clone GitHub repos to the local Gitea during install #212

Clone GitHub repos to the local Gitea during install #212

kispaljr commented Nov 3, 2023 •

edited

nephio-prow bot commented Nov 3, 2023

kispaljr commented Nov 3, 2023

electrocucaracha left a comment

electrocucaracha commented Nov 3, 2023

kispaljr commented Nov 3, 2023

electrocucaracha commented Nov 3, 2023

nephio-prow bot commented Nov 3, 2023

kispaljr commented Nov 3, 2023

kispaljr commented Nov 3, 2023

radoslawc commented Nov 3, 2023

kispaljr commented Nov 3, 2023

electrocucaracha Nov 9, 2023

kispaljr commented Nov 10, 2023

kispaljr commented Nov 10, 2023

kispaljr commented Nov 10, 2023 •

edited

kispaljr commented Nov 10, 2023

kispaljr commented Nov 10, 2023

kispaljr commented Nov 12, 2023

electrocucaracha commented Nov 14, 2023

kispaljr commented Nov 16, 2023

electrocucaracha commented Nov 16, 2023

kispaljr commented Nov 20, 2023 •

edited

electrocucaracha commented Nov 27, 2023

kispaljr commented Dec 1, 2023

kispaljr commented Dec 3, 2023 •

edited

electrocucaracha commented Dec 4, 2023

nephio-prow bot commented Dec 4, 2023

electrocucaracha commented Dec 6, 2023

Clone GitHub repos to the local Gitea during install #212

Are you sure you want to change the base?

Clone GitHub repos to the local Gitea during install #212

Conversation

kispaljr commented Nov 3, 2023 • edited

nephio-prow bot commented Nov 3, 2023

kispaljr commented Nov 3, 2023

electrocucaracha left a comment

Choose a reason for hiding this comment

electrocucaracha commented Nov 3, 2023

kispaljr commented Nov 3, 2023

electrocucaracha commented Nov 3, 2023

nephio-prow bot commented Nov 3, 2023

kispaljr commented Nov 3, 2023

kispaljr commented Nov 3, 2023

radoslawc commented Nov 3, 2023

kispaljr commented Nov 3, 2023

electrocucaracha Nov 9, 2023

Choose a reason for hiding this comment

kispaljr commented Nov 10, 2023

kispaljr commented Nov 10, 2023

kispaljr commented Nov 10, 2023 • edited

kispaljr commented Nov 10, 2023

kispaljr commented Nov 10, 2023

kispaljr commented Nov 12, 2023

electrocucaracha commented Nov 14, 2023

kispaljr commented Nov 16, 2023

electrocucaracha commented Nov 16, 2023

kispaljr commented Nov 20, 2023 • edited

electrocucaracha commented Nov 27, 2023

kispaljr commented Dec 1, 2023

kispaljr commented Dec 3, 2023 • edited

electrocucaracha commented Dec 4, 2023

nephio-prow bot commented Dec 4, 2023

electrocucaracha commented Dec 6, 2023

kispaljr commented Nov 3, 2023 •

edited

kispaljr commented Nov 10, 2023 •

edited

kispaljr commented Nov 20, 2023 •

edited

kispaljr commented Dec 3, 2023 •

edited