Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

boostrap-os: badly formed task file included #11152

Open
Josh-Tracy opened this issue May 1, 2024 · 14 comments
Open

boostrap-os: badly formed task file included #11152

Josh-Tracy opened this issue May 1, 2024 · 14 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Josh-Tracy
Copy link

What happened?

Running the command ansible-playbook -i ./inventory/mycluster/hosts.yaml cluster.yml -K --check -b results in the following output:

BECOME password: 
[WARNING]: While constructing a mapping from /mnt/f/Homelab/kubespray/roles/bootstrap-os/tasks/main.yml, line 29, column 7, found a duplicate dict key (paths). Using last defined value only.
[WARNING]: Skipping callback plugin 'ara_default', unable to load

PLAY [Check Ansible version] ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.262)       0:00:00.262 ********* 

TASK [Check 2.16.4 <= Ansible version < 2.17.0] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1] => {
    "changed": false,
    "msg": "All assertions passed"
}
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.023)       0:00:00.286 ********* 

TASK [Check that python netaddr is installed] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1] => {
    "changed": false,
    "msg": "All assertions passed"
}
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.046)       0:00:00.332 *********

TASK [Check that jinja is not too old (install via pip)] **************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1] => {
    "changed": false,
    "msg": "All assertions passed"
}
[WARNING]: Could not match supplied host pattern, ignoring: kube-master

PLAY [Add kube-master nodes to kube_control_plane] ********************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: kube-node

PLAY [Add kube-node nodes to kube_node] *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: k8s-cluster

PLAY [Add k8s-cluster nodes to k8s_cluster] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: calico-rr

PLAY [Add calico-rr nodes to calico_rr] *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: no-floating

PLAY [Add no-floating nodes to no_floating] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: bastion

PLAY [Install bastion ssh config] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Bootstrap hosts for Ansible] ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.067)       0:00:00.399 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.041)       0:00:00.441 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.025)       0:00:00.467 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.057)       0:00:00.524 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.026)       0:00:00.550 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.057)       0:00:00.607 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.073)       0:00:00.680 ********* 
[WARNING]: raw module does not support the environment keyword

TASK [bootstrap-os : Fetch /etc/os-release] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1]
ok: [node2]
ok: [node3]
ok: [node4]
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.162)       0:00:00.843 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.078)       0:00:00.921 ********* 

TASK [bootstrap-os : Include tasks] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************
fatal: [node1]: FAILED! => {"reason": "included task files must contain a list of tasks"}
fatal: [node2]: FAILED! => {"reason": "included task files must contain a list of tasks"}
fatal: [node3]: FAILED! => {"reason": "included task files must contain a list of tasks"}
fatal: [node4]: FAILED! => {"reason": "included task files must contain a list of tasks"}

NO MORE HOSTS LEFT ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
node1                      : ok=4    changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
node2                      : ok=1    changed=0    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0
node3                      : ok=1    changed=0    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0
node4                      : ok=1    changed=0    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0

Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.097)       0:00:01.019 *********
===============================================================================
bootstrap-os : Fetch /etc/os-release --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.16s 
bootstrap-os : Include tasks ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.10s 
bootstrap-os : Include vars ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.08s 
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.07s 
Check that jinja is not too old (install via pip) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.07s 
kubespray-defaults : Populates no_proxy to all hosts ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.06s 
kubespray-defaults : Set fallback_ips -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.06s 
Check that python netaddr is installed ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.05s 
kubespray-defaults : Gather ansible_default_ipv4 from all hosts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.04s 
kubespray-defaults : Set no_proxy to all assigned cluster IPs and hostnames ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.03s 
kubespray-defaults : Create fallback_ips_base ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.03s 
Check 2.16.4 <= Ansible version < 2.17.0 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.02s

What did you expect to happen?

I expect hosts to be matched and tasks to be included

How can we reproduce it (as minimally and precisely as possible)?

git clone https://github.com/kubernetes-sigs/kubespray.git
pip3 install -r requirements.txt
declare -a IPS=(192.168.50.177 192.168.50.30 192.168.50.228 192.168.50.117)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
ansible-playbook -i ./inventory/mycluster/hosts.yaml  cluster.yml -K --check -b

OS

Ansible Control Node (WSL2)

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Target Hosts (ARM64)

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```bash

### Version of Ansible

```bash
ansible --version
ansible [core 2.16.6]
  config file = /mnt/f/Homelab/kubespray/ansible.cfg
  configured module search path = ['/mnt/f/Homelab/kubespray/library']
  ansible python module location = /home/thesudo/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/thesudo/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

Version of Python

Python 3.10.6

Version of Kubespray (commit)

91dea02

Network plugin used

calico

Full inventory with variables

all:
  vars:
    ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
    ansible_user: pirateking
    ansible_ssh_private_key_file: ~/.ssh/strawhatcrew_key
    
  hosts:
    node1:
      ansible_host: 192.168.50.177
      ip: 192.168.50.177
      access_ip: 192.168.50.177
    node2:
      ansible_host: 192.168.50.30
      ip: 192.168.50.30
      access_ip: 192.168.50.30
    node3:
      ansible_host: 192.168.50.228
      ip: 192.168.50.228
      access_ip: 192.168.50.228
    node4:
      ansible_host: 192.168.50.117
      ip: 192.168.50.117
      access_ip: 192.168.50.117
  children:
    kube_control_plane:
      hosts:
        node1:
        node2:
    kube_node:
      hosts:
        node1:
        node2:
        node3:
        node4:
    etcd:
      hosts:
        node1:
        node2:
        node3:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}

Command used to invoke ansible

ansible-playbook -i ./inventory/mycluster/hosts.yaml cluster.yml -K --check -b

Output of ansible run

[WARNING]: While constructing a mapping from /mnt/f/Homelab/kubespray/roles/bootstrap-os/tasks/main.yml, line 29, column 7, found a duplicate dict key (paths). Using last defined value only.
[WARNING]: Skipping callback plugin 'ara_default', unable to load

PLAY [Check Ansible version] ******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.262)       0:00:00.262 ********* 

TASK [Check 2.16.4 <= Ansible version < 2.17.0] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1] => {
    "changed": false,
    "msg": "All assertions passed"
}
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.023)       0:00:00.286 ********* 

TASK [Check that python netaddr is installed] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1] => {
    "changed": false,
    "msg": "All assertions passed"
}
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.046)       0:00:00.332 *********

TASK [Check that jinja is not too old (install via pip)] **************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1] => {
    "changed": false,
    "msg": "All assertions passed"
}
[WARNING]: Could not match supplied host pattern, ignoring: kube-master

PLAY [Add kube-master nodes to kube_control_plane] ********************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: kube-node

PLAY [Add kube-node nodes to kube_node] *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: k8s-cluster

PLAY [Add k8s-cluster nodes to k8s_cluster] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: calico-rr

PLAY [Add calico-rr nodes to calico_rr] *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: no-floating

PLAY [Add no-floating nodes to no_floating] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched
[WARNING]: Could not match supplied host pattern, ignoring: bastion

PLAY [Install bastion ssh config] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Bootstrap hosts for Ansible] ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Wednesday 01 May 2024  06:35:05 -0400 (0:00:00.067)       0:00:00.399 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.041)       0:00:00.441 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.025)       0:00:00.467 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.057)       0:00:00.524 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.026)       0:00:00.550 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.057)       0:00:00.607 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.073)       0:00:00.680 ********* 
[WARNING]: raw module does not support the environment keyword

TASK [bootstrap-os : Fetch /etc/os-release] ***************************************************************************************************************************************************************************************************************************************************************************************************************************************************
ok: [node1]
ok: [node2]
ok: [node3]
ok: [node4]
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.162)       0:00:00.843 ********* 
Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.078)       0:00:00.921 ********* 

TASK [bootstrap-os : Include tasks] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************
fatal: [node1]: FAILED! => {"reason": "included task files must contain a list of tasks"}
fatal: [node2]: FAILED! => {"reason": "included task files must contain a list of tasks"}
fatal: [node3]: FAILED! => {"reason": "included task files must contain a list of tasks"}
fatal: [node4]: FAILED! => {"reason": "included task files must contain a list of tasks"}

NO MORE HOSTS LEFT ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

PLAY RECAP ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
node1                      : ok=4    changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
node2                      : ok=1    changed=0    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0
node3                      : ok=1    changed=0    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0
node4                      : ok=1    changed=0    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0

Wednesday 01 May 2024  06:35:06 -0400 (0:00:00.097)       0:00:01.019 *********
===============================================================================
bootstrap-os : Fetch /etc/os-release --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.16s 
bootstrap-os : Include tasks ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.10s 
bootstrap-os : Include vars ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.08s 
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.07s 
Check that jinja is not too old (install via pip) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.07s 
kubespray-defaults : Populates no_proxy to all hosts ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.06s 
kubespray-defaults : Set fallback_ips -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.06s 
Check that python netaddr is installed ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.05s 
kubespray-defaults : Gather ansible_default_ipv4 from all hosts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.04s 
kubespray-defaults : Set no_proxy to all assigned cluster IPs and hostnames ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.03s 
kubespray-defaults : Create fallback_ips_base ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.03s 
Check 2.16.4 <= Ansible version < 2.17.0 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.02s 

Anything else we need to know

Additionally, I modified ansible.cfg roles_path provided by the kubespray repo and i have verified that this works as expected.

[ssh_connection]
pipelining=True
ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
#control_path = ~/.ssh/ansible-%%r@%%h:%%p
[defaults]
# https://github.com/ansible/ansible/issues/56930 (to ignore group names with - and .)
force_valid_group_names = ignore

host_key_checking=False
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp
fact_caching_timeout = 86400
stdout_callback = default
display_skipped_hosts = no
library = ./library
callbacks_enabled = profile_tasks,ara_default
roles_path = /mnt/f/Homelab/kubespray/roles
deprecation_warnings=False
inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo, .creds, .gpg
[inventory]
ignore_patterns = artifacts, credentials
@Josh-Tracy Josh-Tracy added the kind/bug Categorizes issue or PR as related to a bug. label May 1, 2024
@Josh-Tracy
Copy link
Author

Through troubleshooting I was able to get beyond these errors by modifying roles/bootstrap-os/tasks/main.yml and specifying the paths manually for the include tasks task:

  - name: Include tasks
    include_tasks: "{{ item }}"
    with_first_found:
    - <<: *search
      paths: 
      - tasks/ubuntu.yml

Obviously this is not desired nor how this should work so I assume that there is a bug or something in my environment causing this?

@VannTen
Copy link
Contributor

VannTen commented May 2, 2024

fatal: [node4]: FAILED! => {"reason": "included task files must contain a list of tasks"}

I think this means the included tasks file is wrongly formatted (the no hosts matched is normal, it's compat stuff for legacy group names)

Can you execute the playbook with more verbosity (-vvvv, each one is one verbosity level more) so we can see what's going on ?

Maybe it's doing something and executing the vars/ files instead ? 🤔 Not sure why that would only happens in ARM64 but you never know...

@Josh-Tracy
Copy link
Author

Attached is the output using -vvvv

log.txt

@VannTen
Copy link
Contributor

VannTen commented May 2, 2024 via email

@VannTen
Copy link
Contributor

VannTen commented May 3, 2024 via email

@k8s-ci-robot k8s-ci-robot changed the title No Hosts Matched While Running cluster.yml boostrap-os: badly formed task file included May 3, 2024
@dc232
Copy link

dc232 commented May 18, 2024

hmm I did this on a whim as I also got the error, the steps below seem to resolve it
try git clone {kubespary_repo}
to cd into the inventory directory in my case, this is
kubespray/inventory/sample

i then ran the created inventory file from the inventory.py file in my case its hosts.yaml

i then set the env var
ANSIBLE_CONFIG=../../ansible.cfg

then I ran the following command
ansible-playbook ../../cluster.yml -i hosts.yaml --ssh-extra-args="-oStrictHostKeyChecking=no" --extra-vars "ansible_user=dc" --private-key="~/.ssh/kubernetes/dc_key" --ask-become-pass --become -vvv

it should be noted that you don't need to have the --private-key= and --extra-vars "ansible_user=dc" flags set I just configured my vm's to have that key and user

i got this as the final output

Saturday 18 May 2024  22:00:26 +0100 (0:00:01.214)       0:09:29.241 **********
===============================================================================
kubernetes/preinstall : Pick coredns cluster IP or default resolver ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 9.93s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/kubernetes/preinstall/tasks/0020-set_facts.yml:168 ----------------------------------------------------------------------------------
download : Download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9.01s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_container.yml:57 --------------------------------------------------------------------------------------------
etcd : Gen_certs | Write etcd member/admin and kube_control_plane client certs to other etcd nodes ----------------------------------------------------------------------------------------------------------------------------------- 7.59s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/etcd/tasks/gen_certs_script.yml:87 --------------------------------------------------------------------------------------------------
kubernetes-apps/ansible : Kubernetes Apps | Start Resources -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.78s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/kubernetes-apps/ansible/tasks/main.yml:30 -------------------------------------------------------------------------------------------
kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS templates --------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.48s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/kubernetes-apps/ansible/tasks/coredns.yml:2 -----------------------------------------------------------------------------------------
container-engine/crictl : Extract_file | Unpacking archive --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.03s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/extract_file.yml:2 ---------------------------------------------------------------------------------------------------
kubernetes-apps/metrics_server : Metrics Server | Create manifests ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.63s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/kubernetes-apps/metrics_server/tasks/main.yml:38 ------------------------------------------------------------------------------------
etcd : Gen_certs | Gather etcd member/admin and kube_control_plane client certs from first etcd node --------------------------------------------------------------------------------------------------------------------------------- 3.62s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/etcd/tasks/gen_certs_script.yml:63 --------------------------------------------------------------------------------------------------
download : Download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.57s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_container.yml:57 --------------------------------------------------------------------------------------------
kubernetes/preinstall : Update package management cache (APT) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.55s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/kubernetes/preinstall/tasks/0070-system-packages.yml:35 -----------------------------------------------------------------------------
container-engine/crictl : Download_file | Download item ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.33s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_file.yml:58 -------------------------------------------------------------------------------------------------
container-engine/nerdctl : Download_file | Download item ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.32s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_file.yml:58 -------------------------------------------------------------------------------------------------
download : Download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.29s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_container.yml:57 --------------------------------------------------------------------------------------------
container-engine/runc : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.25s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_file.yml:58 -------------------------------------------------------------------------------------------------
container-engine/validate-container-engine : Populate service facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.18s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/container-engine/validate-container-engine/tasks/main.yml:25 ------------------------------------------------------------------------
etcdctl_etcdutl : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.18s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_file.yml:58 -------------------------------------------------------------------------------------------------
etcdctl_etcdutl : Extract_file | Unpacking archive ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.08s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/extract_file.yml:2 ---------------------------------------------------------------------------------------------------
container-engine/containerd : Download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.06s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_file.yml:58 -------------------------------------------------------------------------------------------------
download : Download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.87s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/download/tasks/download_file.yml:58 -------------------------------------------------------------------------------------------------
container-engine/containerd : Containerd | Unpack containerd archive ----------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.83s
/mnt/c/Users/danny/Desktop/Terraform_vmware/ansible_scripts/Day_0/kube-spray-bootstrap/kubespray/roles/container-engine/containerd/tasks/main.yml:34 ---------------------------------------------------------------------------------------

and didn't get the warnings about the duplicate key dictionary
I think this is more a quirk with Ansible than anything due to the way group_vars seem to work
i got the idea from this doc
https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html#organizing-host-and-group-variables

hope this helps aid in the debugging

hmm actually upon further tests, the error still persists

@jeanfabrice
Copy link
Contributor

Same issue. Unable to upgrade as the playbook terminates almost immediately.
The workaround shared in #11152 (comment) permitted to move forward.

@VannTen
Copy link
Contributor

VannTen commented May 29, 2024 via email

@jeanfabrice
Copy link
Contributor

I did not run it in check mode as I was applying to a non-production qualification cluster

ansible-playbook -i inventory/dev/hosts playbooks/kubernetes/upgrade-cluster.yml  

is failing the way described by the OP

@aviral-agarwal
Copy link

aviral-agarwal commented May 29, 2024

Hey guys,
Here is a solution I tried and was able to resolve the error, detailing it here
Note: New to Kubernetes, trying to deploy using Kubespray

same issue as Josh-Tracy/how to reproduce

clone kubespray git repo -> cp -rfp inventory/sample inventory/mycluster -> update inventory/mycluster/inventory.ini for nodes -> create python virtual environment and install requirements.txt

execute playbook ansible-playbook -i inventory/mycluster/inventory.ini --become --become-user=root cluster.yml

Checking ansible config from the Python environment

just to be sure
image
As we can see, no change in ansible.cfg or roles_path, tried after updating it but the issue persisted and reverted to the original

About nodes (all nodes same, including the machine I am running Ansible from)

Added new task Debug os_release_dict in kubespray/roles/bootstrap-os/tasks/main.yml
image
image

What is happening

  • when executing kubespray/roles/bootstrap-os/tasks/main.yml for ubuntu nodes, kubespray/roles/bootstrap-os/tasks/ubuntu.yml should be executed
  • file name is dynamically generated during runtime using os_release_dict['ID'] in the Include vars task
  • in the Include tasks task, the include_tasks ansible directive looks for files in the path provided by search YAML anchor-alias,
  • though no match or file is found in the search path
  • but (I believe) include_tasks also automatically looks for files in the same path as the playbook, which it finds,
  • the point is, the file is found, but
    image
    image
    by adding Debug Include tasks, we can see the file is found, but executing it gives an error

Issue (according to me)

  • issue is, the content of ubuntu.yml is just
    image
    this works sometimes but not the way it is currently in the Kubespray GitHub repo and throws the above error

  • this seems to be some kind of quirk of Ansible, that it works when we deliberately provide a path in the search anchor of YAML (as suggested by Josh-Tracy, tried and worked) but when it is automatically found in the same directory by Ansible, proper Ansible YAML file is expected

Solution

update the content of kubespray/roles/bootstrap-os/tasks/ubuntu.yml to proper Ansible task
image
image

Something to think about

this will cause issues if kubespray/roles/bootstrap-os/tasks/ubuntu.yml is being used somewhere else and does not work as expected with this update, which I am not sure about

moving ahead with this for now
Hope this is helpful, Thanks

@VannTen
Copy link
Contributor

VannTen commented May 30, 2024

Ok, so it would seem to be the interaction between symlinks (ubuntu, opensuse-* and openEuler tasks are symlinks) and the first_found lookup ?

total 64
-rw-r--r-- 1 max max  530 May 19 10:43 amzn.yml
-rw-r--r-- 1 max max 4371 May 30 09:16 centos.yml
-rw-r--r-- 1 max max  318 Mar 29 09:32 clearlinux.yml
-rw-r--r-- 1 max max 1912 Mar 29 09:32 debian.yml
-rw-r--r-- 1 max max 1112 Mar 29 09:32 fedora-coreos.yml
-rw-r--r-- 1 max max  920 May  2 16:03 fedora.yml
-rw-r--r-- 1 max max  956 May  2 16:03 flatcar.yml
-rw-r--r-- 1 max max 1652 May 13 13:28 main.yml
lrwxrwxrwx 1 max max   10 May 19 10:43 openEuler.yml -> centos.yml
lrwxrwxrwx 1 max max   12 Mar 29 09:32 opensuse-leap.yml -> opensuse.yml
lrwxrwxrwx 1 max max   12 Mar 29 09:32 opensuse-tumbleweed.yml -> opensuse.yml
-rw-r--r-- 1 max max 2189 Mar 29 09:32 opensuse.yml
-rw-r--r-- 1 max max 4207 May 30 09:16 redhat.yml
lrwxrwxrwx 1 max max   10 Mar 29 09:32 ubuntu.yml -> debian.yml

This is pretty weird, and looks like and ansible bug. We should check if it's a known one.

@VannTen
Copy link
Contributor

VannTen commented May 30, 2024

What's the Filesystem of the ansible controller ? (wondering if this could have an impact)

@jeanfabrice
Copy link
Contributor

jeanfabrice commented May 30, 2024

I think I got it.
My company applied

$ git config core.symlinks    
false

as a countermeasure to git CVE-2024-32004, high

resetting git core.symlinks to default, the symlinks are back as expected and the check mode is passing the bootstrap-os role as expected

@VannTen
Copy link
Contributor

VannTen commented May 30, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

5 participants