Support for pyproject.toml

[rr-]

pypyr seems nice but polluting the filesystem with a few small files is the opposite of what I want from this tool, which is to replace makefiles such as this one:

test:
	pytest --cov=dict --cov-report term-missing

lint:
	pre-commit run -a

clean:
	rm .coverage
	rm -rf .mypy_cache
	rm -rf .pytest_cache

.PHONY: test lint clean

and on top of it, allow me to pass extra arguments to it so I can make test my/test.py, and use the now-standard pyproject.toml for it (just like setuptools has setup.py and npm has package.json).

[yaythomas]

thanks for the feedback @rr-!

Could you please help me understand better, because I'm not entirely clear on what you're suggesting? Do you mean that you would not like to have yaml pipeline files and you instead want to define your step sequences in pyproject.toml itself?

[rr-]

Yes

[rr-]

Just like for example we can define pylint settings in pyproject.toml across multiple sections, I imagine each pipeline would be a separate section, key of which begins with the same prefix. such as [pypyr-test]

[yaythomas]

Thanks for explaining some more, @rr-, that really helps! 🙌

There are a few things that make putting the entire pipeline into pyproject.toml difficult:

1. toml is a deliberately minimal configuration file format, so toml doesn't allow some of the yaml language structures that a pypyr pipeline relies on - specifically tagging for custom types. This means !py strings !jsonify etc won't translate directly to toml.
2. pypyr pipelines can get pretty complex - especially as you start sequencing not just CI but also more tricky CD functionality - and therefore the pipelines can get pretty long/big. I don't think pyproject.toml is really meant to encode business logic like this, because it is a declarative configuration file, not a piece of code -
   a) firstly because of churn/contention on the pyproject.toml file if multiple pipelines exist in the one file (different people might be working on different pipelines) - you prob don't want churn on your proj toml file for devops/gitops deployment updates, for example.
   b) the "flatter" toml format will make pipelines like this hard to read for a human (okay, hard for me, maybe someone who reads toml for a living might think differently 😅 ),
   c) the pyproject.toml file will get big and unwieldy. pypyr pipelines can go to 100s-1000s lines. So on the one hand you can say pypyr pipelines "pollute" the file-system, but on the other hand you could say that all this business logic would "pollute" pyproject.toml. Which is the better option, I don't know!
3. yaml is a bit more forgiving on not having to quote/escape strings - and this is actually something that makes pypyr yaml a lot easier to deal with when you're expressing things where literal quotes matter like coding statements with quotes print("hello's") inside a pipeline. I think transcribing this type of logic into toml with string escaping is going to be annoying for the author.
4. pypyr has a lot of use-cases and current real-world usage outside of simple python project automation - so it's not the case that all of those will already have a pyproject.toml file. (I can think of at least one proj that stores pipelines in ZooKeeper, not on the filesystem) This means pypyr would have to maintain 2 different ways of parsing pipelines - toml and yaml. And achieving feature parity with toml is not as simple as just adding a toml parser, because of the missing language structures in toml.
5. part of what makes pypyr pipelines useful is that you can create shared pipeline repositories, for modular + composable functionality where pipelines can consume other ready-made pipelines. If you first have to copy/mangle what you want into a project toml file, you lose that component reusability.

Now, having said some of the reasons not to do it. . .

Your idea and the example you give looks sensible and I like your suggestion very much! IF what we're talking about is very static, unlikely to change, simple one-liner commands that you pretty much just set at the beginning of the project and then never touch again. In there words, like you show in your example.

So if you had pyproject.toml like this:

[tool.pypyr]

[tool.pypyr.commands]
test = "pytest --cov=dict --cov-report term-missing"
lint = "pre-commit run -a"
clean = """
  rm .coverage
  rm -rf .mypy_cache
  rm -rf .pytest_cach
"""
# use list to run multiple commands
all= ["test", "lint", "clean"]

And then you'd run like:

$ pypyr test
$ pypyr lint
# as per make style, run 2 targets from the cli
$ pypyr lint clean
$ pypyr all

I'll have to think a bit more about how or if arguments + variables can be made to work, so that:

[tool.pypyr.commands]
test = "pytest --cov=dict --cov-report term-missing ${f}"

You can run:

$ pypyr test f=./myfile.py
# or maybe even, but this might not be possible to do cleanly. . .
$ pypyr test ./myfile.py

So this would basically be a simplified "command only" or "array of command" syntax that you can set in your pyproject.toml - you don't have access to the more heavy logic of a full pypyr pipeline like built-in steps/looping/conditional branching/auto-retries etc. So while I'm hesitant about trying to force "full" pipeline functionality into pyproject.toml, what we're talking about here instead pypyr is really just processing a sequence of one-liner commands with a makefile style TARGET syntax.

What do you think, @rr-? Would something like this help you, or do you have suggestions for doing it differently, or does it sound like a can of worms you don't even want to open 😆 ?

[yaythomas]

Oh also, something extra I think would be useful is to create a pypyr built-in step to load configuration values from pyproject.toml.

So you would have your yaml pipeline to encode your business logic flow. And then the variable configuration values you can store in pyproject.toml which is exactly what's supposed to go into pyproject.toml, project specific declarative configuration values. . .

So these config values can then load into the pipeline at runtime with a ready-made step.

[rr-]

I read your comment carefully. Thanks for the reply.
This flat command structure is exactly what I need. I don't think I want full-fledged pypyr pipelines, especially since, as you mentioned, they can get very large. It would also be good to be able to pass any arguments transparently, without the need to describe them at the command declaration level, so that e.g. I can do:

pypyr test -v3 -s myproject/tests/test_views.py --pdb

without need to declare -v, -s, the path argument or --pdb. This could be done by forcing $(args) to be specified in the script body, so that pypyr knows exactly which command is to take these additional user arguments.

In general, I'm not sure pypyr is a good tool to implement this feature. As I wrote, I just wanted to replace make with some tool that 1) allows you to pass arguments to subcommands 2) does not require cluttering the project directory with new files and instead uses pyproject.toml, which already allows you to configure tools such as black, isort, mypy, pylint, flake8 etc. I couldn't find any tool that would offer such a thing, so I had two options: either start a new project (cool, but litters the ecosystem with yet another package for something rather basic) or find an existing project with sufficiently similar goals, and ask for the implementation of this feature there.

Now that I think of it, it would probably also be nice to be able to install shell autocomplete that at least reads the commands (pypr t → pypyr test). It sounds like something bigger and I don't know if it would align well with the pypyr's goals.

[yaythomas]

Yes, what you're describing with transparent args sounds very useful and I'd find it handy myself! I'll have to think a bit about how best to go implementing some more.

Re what you're saying whether pypyr is necessarily the best tool for this. . . You make a good point! The simple scripts section in package.json that you can $ npm run <user defined> from the cll are, of course, great for relatively simple commands. . . but they're not really meant for more difficult/complicated workflows - even if you're just doing some non-trivial js transcoding/tree-shaking/asset processing/differential builds/dependency tracking you're likely to use a task-runner like gulp or grunt to do the actual work.

In the case of make, it is an amazing build utility. Because it such a well-built & useful tool it so happens over the years (decades!) people have also been using it as a general purpose automation utility or just as a simple "task runner" with phony targets, which often doesn't really use the actual functionality that makes make great (e.g dependency tracking for quicker rebuilds).

It so happens pypyr exists because over and above the pure "build" functions of a project, for the broader automation functions like packaging & deployment (CD) & devops automation workflows, npm scripts or make are not really meant for this and can lead to frustration when you're trying to start using them as general purpose pipeline or workflow automation tools. pypyr is not at all particularly about tracking dependencies/selective build targets - use make or gulp or similar for that. Instead, a pypyr pipeline might well invoke make or an npm run install to do the building as a first step, let those tools do what they're good at, and then run additional steps to upload the output somewhere, retry web-service calls if they fail, automatically spin up a testing rig somewhere etc.

TOML parsing in general is definitely coming to pypyr (#134) - and I am planning to add a convenience step to read pipeline config from pyproject.toml in a single easy step. That way a pypyr pipeline can consume changeable configuration data that exists in pyproject.toml. This would allow you to use a pypyr pipeline to iterate over & "run as command" strings stored in pyproject.toml.

This is not quite what you're describing, however, because you'd still need a pipeline - although I have half a mind to create a built-in pipeline to do just this - loop over a [tool.pypyr.commands] section in the current directory's pyproject.toml and execute them.

In this case you have

$ pypyr do-pyproject-toml-scripts -- test -v3 -s myproject/tests/test_views.py --pdb

(I'd have to think of a better pipeline name than do-pyproject-toml-scripts 😆 Ideas welcome!)

Since the prospective do-pyproject-toml-scripts pipeline would be built-in to pypyr, you wouldn't have it cluttering up your project directory. Doing it this way would actually be relatively easy for pypyr. Trying to implement it without a pipeline is going to be harder and maybe going against the grain of what pypyr is good at.

Some suggestions:

I'm sure you know about these already, but just for the sake of completeness, I'd feel remiss not to point out some other excellent open-source projects in this arena. . .

You might want to check out tox - this is an amazing tool that not only manages your virtual environments for you, but allows you to specify commands with a "passthrough" argument style like you want! However, tox thinks more in terms of "environments" than "targets", but you could more or less make it behave similarly to the makefile example you gave if you treated each target as an environment in tox.

Since pyproject.toml is relatively new, there is a whole new build ecosystem burgeoning around it - poetry & flit being the obvious examples. These are focused on build, not so much the task-running.

You might want to check out https://github.com/nat-n/poethepoet - I've not used this myself (yet), but it looks like a great project! It does most (or maybe some) of what you're after, especially if you're in the poetry ecosystem already!

[rr-]

Thanks for this reply.

I'd have to think of a better pipeline name than do-pyproject-toml-scripts laughing Ideas welcome!

Maybe it could do without the command and try to match the test against any tool.pypyr.commands as a fallback mechanism?

[yaythomas]

Yep, so not having the "command" (aka a pipeline name) so you can just go $ pypyr blah where blah is defined in tool.pypyr.commands is going to be quite a bit more tricky to implement and somewhat against the grain of how it currently works - but of course, not that something being difficult is a good reason for not doing it! But I am thinking about this and it might well become possible because of another change request I'm working on to provide "pipeline shortcuts" in a config file.

But if we're going to stick to the easier way of doing this (for now), If we take npm as example, if package.json

{
  "scripts": {
    "start": "node index.js",
    "dothing": "mycommand --blah",
  }
}

You can then execute:

$ npm start
$ npm run dothing

So start is a special reserved keyword, so you don't have to type run - but for other/arbitrary custom scripts you still need to type npm run <<mything>>.

So similarly I guess, instead of "do-pyproject-toml-scripts", maybe

$ pypyr run dothing

where dothing is defined in [tool.pypyr.commands] in pyproject.toml?