Recipies for for analyzing python dependencies with common "pip-adjacent" tools

[cburroughs]

pip-licenses, pip-audit , and pip itself are useful tools for analyzing Python dependencies, but they don't know how to read a PEX lockfile. You can pants export and run any of these, but I took a rough stab at making them one line targets.

pex_binary(
    name="the-big-pex",
    include_tools=True,
    dependencies=[
    # [python.resolves]
    # default = "3rdparty/py/default.lock"
        "3rdparty/py",
    ],
)

adhoc_tool(
    name="adhoc-pip-freeze",
    runnable=":the-big-pex",
    extra_env_vars=["PEX_SCRIPT=pip"],
    args=["--disable-pip-version-check", "freeze"],
    root_output_directory=".",
    stdout="freeze.out",
)

run_shell_command(
    name="pip-freeze",
    execution_dependencies=[":adhoc-pip-freeze"],
    command="cat {chroot}/freeze.out",
)

adhoc_tool(
    name="adhoc-pip-list-outdated",
    runnable=":the-big-pex",
    extra_env_vars=["PEX_SCRIPT=pip"],
    args=["--disable-pip-version-check", "list", "-o"],
    root_output_directory=".",
    stdout="list.out",
)

run_shell_command(
    name="pip-list-outdated",
    execution_dependencies=[":adhoc-pip-list-outdated"],
    command="cat {chroot}/list.out",
)

adhoc_tool(
    name="adhoc-pip-licenses",
    runnable=":the-big-pex",
    extra_env_vars=["PEX_SCRIPT=pip-licenses"],
    args=[],
    root_output_directory=".",
    stdout="licenses.out",
)

run_shell_command(
    name="pip-licenses",
    execution_dependencies=[":adhoc-pip-licenses"],
    command="cat {chroot}/licenses.out",
)


adhoc_tool(
    name="adhoc-pip-audit",
    runnable=":the-big-pex",
    execution_dependencies=[":adhoc-pip-freeze"],
    extra_env_vars=["PEX_SCRIPT=pip-audit"],
    workdir="/",
    args=["--no-deps", "--disable-pip", "-r", "{chroot}/freeze.out"],
    root_output_directory=".",
    stdout="audit.out",
)

# NOTE: pip-audit insists on returning zero which incidentally prints to
# useful information to the console but doesn't give a clean report file.
run_shell_command(
    name="pip-audit", execution_dependencies=[":adhoc-pip-audit"], command="cat {chroot}/audit.out"
)

[cburroughs]

See also #16495

[huonw]

Nice one!

On the surface, I think using the adhoc_tool target for these sort-of "check external state" checks may lead to misleading results because they can be cached. So, for instance:

1. first run of pip-list-outdated
2. a dep releases an update to PyPI
3. second run of pip-list-outdated, on the same machine and with with no changes to the command inputs (for instance, the dependency lock file)

I think step 3's result will be served from the cache from step 1, the process won't run, and the new dep version in step 2 won't be detected.

[cburroughs]

Yeah I think that's right. I struggle with what the right cache semantics ought to be here. I think it's a better fit for a TTL style cache if that was an option. (I also struggled with using run_shell_command directly with an equivalent to extra_env_vars, but that might have a more straightforward fix.)

[huonw]

We discussed something vaguely similar to this on Slack and @cburroughs noted that a possible solution there might help with this: https://pantsbuild.slack.com/archives/C046T6T9U/p1709941336405819

1. In the .pants.bootstrap file set an env var like export HACK_INVALIDATE_OFTEN=$(date) that changes on basically ever run (name is customisable, of course)
2. Have targets depend on this env var (but do nothing with it), e.g. extra_env_vars=["HACK_INVALIDATE_OFTEN"] if the target supports that, or somehow feeding env("HACK_INVALIDATE_OFTEN") into it from the BUILD file