pickle-spree

Pre-load and run pickled callables in Python subprocesses

What does this do?

It provides a class PopenFactory whose instances constitute replacements for subprocess.Popen¹ which check if the process to be executed is a Python process; if so, it pickles² a given callable, then loads and runs it inside the child Python before any other code is executed in it.

¹ And thereby other parts of subprocess that use Popen internally, such as run.

² Actually dilled to handle some edge cases better.

Why would I need this?

The use case this was written for is convenient monkeypatching within a child Python process, e.g. to patch in mocks for testing when the tested library itself spawns Python subprocesses as part of its operation.

Example

Here is an example that monkeypatches sys.stdout in a Python subprocess so that printed messages are redirected to a file:

from pickle_spree import PopenFactory

from pathlib import Path
import subprocess
import sys

class SysStdoutRedirectingPatch:
  """
  Callable that patches sys.stdout to redirect writes to the given file
  """
  def __init__(self, redirect_to):
    self.redirect_to = redirect_to

  def __call__(self):
    def redirected_write(s):
      with open(self.redirect_to, "a") as f:
        f.write(s)
    import sys
    sys.stdout.write = redirected_write

if __name__ == "__main__":
  patch = SysStdoutRedirectingPatch("output.txt")
  Path("output.txt").unlink(missing_ok=True) # delete previous output file

  # construct Popen replacement and tell it which callable to pickle for
  # loading and execution in the Python subprocess
  new_popen = PopenFactory(callable=patch, pickled_path="dbg.pickle")

  # manually monkeypatch subprocess.Popen; for greater convenience, you could
  # use a library like https://github.com/Plexical/altered.states
  old_popen = subprocess.Popen
  subprocess.Popen = new_popen

  # write script to run in child Python subprocess
  Path("child_script.py").write_text("print('Hello world')")

  # run subprocess using the same executable as the current Python process
  # (usually guarantees being able to load pickled data)
  subprocess.run([sys.executable, "child_script.py"], check=True)

  # check whether everything worked and output was actually redirected:
  output_from_file = Path("output.txt").read_text()
  print(f"output found in output.txt: {output_from_file}") # 'Hello world'

  # undo monkeypatching (again, better use something like altered-states)
  subprocess.Popen = old_popen

Installation

pip install git+https://gitlab.com/smheidrich/pickle-spree.git

API

As the code is extremely short, just have a look at the docstrings there.

Limitations

• Callables must obviously be picklable (but as we use dill, the requirements are a bit more lax, see here) and the pickled data must be loadable by the child Python interpreter.
• Normally, the 2nd requirement entails that all modules defining classes, functions etc. which are used inside the pickled data are importable. However, we use dill instead of pickle, which writes any definitions it encounters within the "main" script into the pickle file (TODO: what about other modules?), so you don't have to do anything to ensure that this one can be imported, which would be the trickiest bit. It's enough to ensure that any modules the main script itself imports can be imported by the child Python interpreter.
• The full range of command-line arguments for child Python interpreters is not available. Only those supported by imphook, which are -m module and name_of_your_script.py, will work. Sorry about that, could be fixed in theory but I don't need it right now.

Related projects

• pymonkey (inactive): Also meant to allow pre-patching in Python processes, but doesn't do any automatic pickling. In principle, pymonkey could be used as a "backend" for pickle-spree that takes care of the pre-execution, but I currently (ab)use imphook for that as I find it easier to understand.