Automated build process generation exploration

[danielrbradley]

This PR contains a comparible implementation of the file-target based makefile in taskfile. It also contains a prototype of how we could generate makefiles by building simple funtions (like we do for our yaml GH actions).

Considering the elements of cross-platform, up-to-date target tracking, the ability to centrally generate and parallelism; there's currently no selling point strong enough for us to shift away from make.

Make

The makefile in awsx is already in a very good place - it uses file targets which means it's trivial for make to optimize which targets need to run based on any file in the project changing. This gives us a good baseline to compare against taskfile.dev.

Taskfile

Overall, taskfile offers very similar features to make with a yaml based language which would be simpler to generate.

Factfile (true as of 3.12.1)

• For checking up to date files, Taskfile uses file hashes by default but also supports timestamp.
• If using timestamps, you have to specify "generates" as well as "sources" for each target.
• If "generates" is specified, it also triggers the task if the files listed are missing.
• Source globbing doesn't support exclusions.
• No alternative to make's --touch to track back through dependencies and mark as built.
• The file yaml format uses more lines of code.
• Dependencies are moddled separately to sources (which are the same thing in make)
• Concurrency attributes allow a clean to be run correctly in parallel at the same time as a build (unlike make).
• The concurrency attributes must be specified on every target to behave correctly.
• Though the CLI tool runs cross-platform, the shell commands within the taskfile are still platform dependant.

The lack of --touch alternative is a blocker for efficient (& correct) CI. When running in CI we want to:

1. Run the first half of the build in the first job
2. Upload the binary assets of the build result
3. Run parallel jobs which download the binary artefacts
4. We use --touch after restoring the artefacts so make will create stub files to make the dependencies appear up-to-date
5. When the second half of the build runs, the existing binaries will be used.

All targets always self-describe all prerequisites so In local situations where we don't use --touch dependencies will be followed as usual ensureing no internal knowledge of the build process for end-users.

Makefile generation

If we can't use taskfile due to the CI limitations, we can instead just generate makefiles dynamically using a quick DSL (< 80 LOC).

Demo program:

import { writeFileSync } from "fs";
import { Makefile, render, Target } from "./makefile";

// Write functions to parametrize targets
function myTarget(arg: string): Target {
  return {
    name: `bin/${arg}`, // Use file targets internally
    commands: [`build ${arg} bin/${arg} $(VERSION)`], // Mix string interpolation and make variables
  };
}

const buildMyProgram = myTarget("my-program"); // Create a target

const makefile: Makefile = {
  variables: {
    // Define make variables
    VERSION: "$(shell pulumictl get version)",
  },
  targets: [
    buildMyProgram, // Add targets to the makefile model
    {
      name: "build",
      dependencies: [buildMyProgram], // Depend on other target objects
      phony: true, // Inline phony attribute
    },
  ],
};

const source = render(makefile);
writeFileSync("Makefile", source);

Output:

VERSION := $(shell pulumictl get version)

bin/my-program:: 
	build my-program bin/my-program $(VERSION)

build:: bin/my-program

.PHONY:: build