Minion with multiple network interfaces fails to connect to master - [ERROR ] No master could be reached, Unable to sign_in to master

[SS67]

Description of Issue

My salt-master setup works seamlessly with single interface VMs, but it doesn't against group of VMs with multiple interfaces.
It appears that minion is unable to pick a gateway/route to send public key the master IPs.

Additional interfaces are for network storage!!

It's a multi-master topology. lets say nefario01 has <ip 1> and nefario02 has <ip 2>
Lets call minion under question - minionbob.example.com

Currently I point minions to master using /etc/salt/minion.d/master.conf
master:

 - <ip 1>
 - <ip 2>

I think in case of multiple interfaces, it need some additional directives in /etc/salt/minion file but I am not sure about them.
Until now I have tried IPTABLES exception for ports 4505/06

Directives I have tried in /etc/salt/minion :

master: 
 - <ip 1>
 - <ip 2>

master_type: failover
ipv6: False
retry_dns: 0

mine_functions:
  network.ip_addrs:
    interface: ens192
    cidr: '0.0.0.0/0'

ens192 is the main host IP interface

Setup

Debug log from minionbob

[root@minionbob ~]# salt-call -d -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: minionbob.example.com
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/master.conf
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'package' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/zfs.py'>
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Using selector: EpollSelector
[INFO    ] Got list of available master addresses: ['<ip 1>', '<ip 2>']
[DEBUG   ] Master URI: tcp://<ip 1>:4506
[DEBUG   ] Master URI: tcp://<ip 2>:4506
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] Master URI: tcp://<ip 1>:4506
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'minionbob.example.com', 'tcp://<ip 1>:4506')
[DEBUG   ] Generated random reconnect delay between '9677ms' and '19677ms' (9677)
[DEBUG   ] Setting zmq_reconnect_ivl to '9677ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '19677ms'
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Closing AsyncReqChannel instance
[INFO    ] Master <ip> could not be reached, trying next master (if any)
[WARNING ] Master ip address changed from <ip 1> to <ip 2>
[DEBUG   ] Master URI: tcp://<ip 2>:4506
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', '.example.com', 'tcp://<ip 2>:4506')
[DEBUG   ] Generated random reconnect delay between '1861ms' and '11861ms' (1861)
[DEBUG   ] Setting zmq_reconnect_ivl to '1861ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11861ms'
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Closing AsyncReqChannel instance
[INFO    ] Master <ip 2> could not be reached, trying next master (if any)
[ERROR   ] No master could be reached or all masters denied the minion's connection attempt.
Unable to sign_in to master: Attempt to authenticate with the salt master failed with timeout 

Steps to Reproduce Issue

• Spin up a VM (minion)
• attach 3 vLANs
• Create interfaces with single gateway
• ensure it connects with salt repo, master etc
• try to report this minion to salt-master

Versions Report

Master version report

[root@nefario01 ~]# salt --versions-report
Salt Version:
          Salt: 3007.0

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: 1.7.2
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.14.1
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Extensions:
        SSEAPE: 8.16.2.2

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: rhel 8.8 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-477.21.1.el8_8.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.8 Ootpa

=================================================================

minion version
[root@minionbob ~]# salt-call --version
salt-call 3007.0 (Chlorine)

=================================================================

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[alrf]

@SS67 check SELinux status and try to disable it - in my case it worked: #66438 (comment)

[SS67]

sestatus shows disabled

[SS67]

I see a common error on all VMs having multiple interfaces:
alt-minion[36457]: salt.exceptions.SaltClientError: Unable to sign_in to master: Attempt to authenticate with the salt master failed with timeout error

[SS67]

It was vLAN route/firewall policy issue. Issue resolved.