[BUG] non operational returner configured as master_job_cache causes salt cli failures (3007.0)

[ITJamie]

Description
running any salt cli commands fails on salt master 3007.0 if master_job_cache: postgres is configured (along with relevant config)

Setup
in /etc/salt/master.d/eventdb.conf

master_job_cache: postgres
returner.postgres.host: 'localhost'
returner.postgres.user: 'username'
returner.postgres.passwd: 'password'
returner.postgres.db: 'salt'
returner.postgres.port: 5432

Steps to Reproduce the behavior
run any salt command. eg. salt * test.ping
response comes back of

'str' object has no attribute 'pop'

Expected behavior
output to happen
Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.0

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: 1.7.2
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.14.1
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.15.0-1061-azure
        system: Linux
       version: Ubuntu 20.04.6 focal

[ITJamie]

got a chance go reduplicate this on a test instance. found the following journal logs.

salt-master[5572]: [ERROR   ] Failed to allocate a jid. The requested returner 'postgres' could not be loaded.
salt-master[5572]: [ERROR   ] Some exception handling a payload from minion
salt-master[5572]: Traceback (most recent call last):
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/server.py", line 181, in handle_message
salt-master[5572]:     ret, req_opts = yield self.payload_handler(payload)
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 767, in run
salt-master[5572]:     value = future.result()
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 1196, in _handle_payload
salt-master[5572]:     ret = await self._handle_clear(load)
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 1240, in _handle_clear
salt-master[5572]:     reply = await method(load)
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 2473, in publish
salt-master[5572]:     payload = self._prep_pub(minions, jid, clear_load, extra, missing)
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 2573, in _prep_pub
salt-master[5572]:     self.event.fire_event({"minions": minions}, clear_load["jid"])
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 769, in fire_event
salt-master[5572]:     event = self.pack(tag, data, max_size=self.opts["max_event_size"])
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 440, in pack
salt-master[5572]:     salt.utils.stringutils.to_bytes(tag),
salt-master[5572]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/stringutils.py", line 53, in to_bytes
salt-master[5572]:     raise TypeError("expected str, bytes, or bytearray not {}".format(type(s)))
salt-master[5572]: TypeError: expected str, bytes, or bytearray not <class 'dict'>

I eventually tracked it down to an issue with psycopg2. I force re-installed psycopg2-binary and that got it working.

it's not great that a broken returner breaks the cli in this way though