A structure for common tutorial scenarios

[blaisep]

I drafted a gist at https://gist.github.com/blaisep/0b8ddf61dcbf6937a1a6c0c2c394c4d3
to try to describe a scenario and how one might approach it.
Perhaps it could be an additional section to https://pypyr.io/docs/pipelines/

I know if might sound like "Automation for dummies" but I suspect many of us don't know how to structure our pipelines if we haven't done it before.

[yaythomas]

Hi @blaisep!

Yes, pypyr is light (ahem, where light=non-existent) on too many practical hands-on step-by-step guides focused on a particular task. So your colophon (wow, fancy word!) is entirely correct.

The Getting Started section does have a "write your 1st pipeline" sort of tutorial, and I like to think throughout each step's documentation page I generally try to pick examples that're hopefully useful/practical. . . but your point is very valid - there isn't a How To/Walkthrough section that aims to go in a more "step by step reference implementation" style examples.

I am more than enthused adding some! I have actually considered this before, but in between writing the code, planning next features, dealing w tech debt (just upgraded from deprecated setuptools to flit for examples, which is not even something a casual end-user should notice) and writing the explanation/reference docs I haven't had that much time.

So if you're volunteering. . . welcome aboard 😁

Couple of notes on your gist:

1. You don't need run: !py cmdOut.returncode == 0 on the step after the 1st pieman machine start cmd - pypyr automatically quits the pipeline reporting failure if a cmd step returns anything except 0.
2. You don't need the single quotes around the multiline podman command - the folding > yaml indicator tells the parser what follows is string, not structural yaml.
3. "Is the Launch not yet complete?" a pattern I use here is to use retry to keep on retrying a command until it's successful. For example:

  - name: pypyr.steps.cmd
    description: --> installing just published release from pypi for smoke-test
    retry:
      max: 5
      sleep: 10
    in:
      cmd: pip install --upgrade --no-cache-dir {package_name}=={expected_version}

In this example, pypyr will retry the pip install command to install the specific version up to 5 times, with 10 seconds sleep in between. Once the pip command returns status code 0, pypyr automatically continues to the next step. I use this when publishing new packages to the pip, because sometimes pip takes a couple of minutes before the newly uploaded version is available for download.

So if you can craft a command that checks the state of the VM and returns a 0 when you're ready to continue, you can follow the same pattern.

4. Idempotency + complex command

There are various ways you can go about achieving idempotency. . . Like you describe, yes, you can generate some sort of unique name on each invocation.

Instead, I have a suspicion you might be better served by using a known name - let's say "api-server", and then making it so that only your pipelines are supposed to use that name. Then make sure as part of your pipeline that you:
a) check the it's not already running. if it is. . . stop it and start a new instance
b) do your work
c) clean-up after fail (stop the instance).

So putting that all together:

steps:
  - name: pypyr.steps.call
    in:
      call: start-services

  - name: pypyr.steps.echo
    in:
      echoMe: do your work here and in subsequent steps once services started

  - name: pypyr.steps.call
    comment: stop the services when done. could also put this in `on_success` group.
    in:
      call: stop-services


start-services:
  - name: pypyr.steps.cmd
    comment: this cmd should return 0 if podman api-server already running
    swallow: True
    in:
      cmd: echo some sort of cmd to check if api-server is running

  - name: pypyr.steps.call
    comment: if previous step errored, means podman already running.
             so stop it first, so we can start clean.
    run: !py "'runErrors' in locals()"
    in:
      call: stop-services

  # if stop is asynchronous/detached, you might have to have another retry-style
  # step here to wait for it to stop completely.
  
  - name: pypyr.steps.cmd
    comment: now we know api-server is deffo NOT running. so can just start it here.
    in:
      cmd: echo start your api-server 

  - name: pypyr.steps.cmd
    comment: this will retry max 5 times w 10s sleep in between to wait for api-server to start
             if api-server does not start in this time, will raise error and stop here.
    retry:
      max: 5
      sleep: 10
    in:
      cmd: echo some sort of cmd that returns 0 when api-server started and ready

stop-services:
  - name: pypyr.steps.cmd
    in:
      cmd: echo stop services here podman api-server stop etc.

# this is the global err handler. this on_failure group will run if any unhandled
# (i.e un-swallowed) error in the pipeline happens.
on_failure:
  - name: pypyr.steps.call
    comment: try to stop the services. this is best effort, so if this fails
             just swallow, not much more we can do.
    swallow: True
    in:
      call: stop-services

Here I've made the assumption that you want to start w a fresh new VM each time. You could alter the above so if it's already running it'll just use the running instance instead of stopping it and restarting it 1st, depending on exactly what you want to achieve.

[edit:] P.S throughout I just used echo in each cmd to describe in words what the actual cmd should be doing, conceptually speaking. I didn't bother looking into the exact podman commands myself :-)

[edit:] P.S you don't "have" to use call to call start-services and stop-services in your pipeline, you could just inline the steps in the main steps sequence if you wanted to. . . but if there are multiple steps in start/stop (e.g waiting for a service to be ready, waiting for it to stop entirely) then it might be better to achieve re-use and composability to encapsulate these in their own re-usable groups like this.

[blaisep]

Ok, I will use these excellent suggestions to update that gist and if you like where I'm going with it, I'm happy to submit a PR.
I imagine there could be a section adjacent to https://pypyr.io/docs/pipelines/pipeline-structure/#pypyr-pipeline-format
called "anatomy of a pipeline" with some general examples of use cases, and possibly links to more detailed ones in the example repo....

btw, I really appreciate the distinction of description vs. comment:
https://pypyr.io/docs/decorators/description/

[blaisep]

@yaythomas , I suppose this could become a tutorial in response to https://pythonspeed.com/articles/shell-scripts/