[BUG] salt-call grains module doesn't have access to pillar

[jtraub91]

Description
What's particularly weird about this issue is that salt-call does have access to pillar when running salt-call saltutil.sync_grains and/or saltutil.refresh_grains but doesn't when retrieiving the grain, e.g. salt-call grains.item or salt-call grains.get. Using pillar in the __virtual__ function as shown below, this results in not being able to retrieve the custom grain with salt-call.

Setup

Install salt-master and salt-minion on a VM. Configure with the following files.

# /etc/salt/master.d/master.conf
user: root

# /etc/salt/minion.d/minion.conf
master: localhost
log_level_logfile: info

# /srv/salt/_grains/test_grain.py
import logging

log = logging.getLogger(__name__)

__virtualname__ = "test_grain"

def __virtual__():
    if __pillar__.get("test"):
        log.info("has test pillar")
        return __virtualname__
    log.info("doesn't have test pillar")
    return (False, "doesn't have test pillar set to True")

def test_grain():
    log.info("test_grain")
    return {"test_grain": "abc123"}

# /srv/pillar/top.sls
base:
  "*":
    - test

# /srv/pillar/test.sls
test: True

Restart the minion and master, as necessary, and accept the minion key. Refresh pillar and sync_grains, as necessary.

Steps to Reproduce the behavior

After syncing grains via saltutil.sync_grains observe that you are able to retrieve the grain with

salt \* grains.item test_grain

but not with

salt-call grains.item test_grain

From the log statements included in the module, we can see in the logs that the pillar is not available when using salt-call grains.item, but is when calling salt-call saltutil.sync_grains (but looks like it's evaluated twice in the latter case and maybe why??)

Expected behavior
salt-call should behave the same as with the salt command in its ability to retrieve custom grains even when pillar is used in __virtual__

Versions Report

Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-67-generic
        system: Linux
       version: Ubuntu 22.04.2 jammy

[max-arnold]

I think it is a chicken and egg problem. Grains resemble facts about the target system and are often used to assign pillar data. Doing the reverse (using pillars to calculate grains) can cause circular dependencies. Something has to be first, and grains behave exactly like that.