Add postProcess plugin hook

[lydell]

Background

In one of my projects I've added the following to my config:

overrides:
    production:
        plugins:
            afterBrunch: [
                # Cache bust.
                "hash-filename *.css *.js *.png *.jpg > #{map = 'map.json'}"
                # Update paths in HTML and CSS to cache busted paths.
                "map-replace --match '<[^>]+>' *.html < #{map}"
                "map-replace --match 'url\\([^)]+\\)' *.css < #{map}"
                # Compress.
                "zopfli *.js *.css"
                ]
                .map((command)-> "cd public/ && #{command}")

I use the after-brunch plugin to run the following command line tools, to achieve cache busting
and gzip compression for production:

• [hash-filename](npm install -g hash-filename)
• [map-replace](npm install -g map-replace)
• zopfli

That works fine for me in this case, where my public/ directory contains no subdirectories. If it would, though, it wouldn't be so easy to specify files for the commands anymore, in a way that works cross-platform.

I've developed the hash-filename and map-replace programs myself. I initially intended them to be brunch onCompile plugins, but in the end I found this solution much easier to program (and I needed such tools for more than brunch, so I killed two birds with one stone).

But I've read many brunch issues about cache busting and gzip compression. I guess there should be brunch plugins for that, since it seems to be a great demand for it.

There actually is a gzip plugin (gzip-brunch), but just like most other onCompile plugins, it suffers from a few things.

Issues

onCompile plugins issues:

• If you want to chain them, they have to do everything synchronously. In my case, hash-filename must be run before map-replace. Luckily, it is possible to hash synchronously, but not all modules have sync interfaces.
• Each plugin has to read and write files. Doing everything in memory would be faster, wouldn't it? At least a lot easier. And no messing with readDirSyncRecursive(...).

So, a change is needed, both for my use case, and for onCompile plugins in general.

Proposal

Change onCompile: (generatedFiles) -> to onCompile: (public, callback) ->, where public is an array of files. A file is an {path: ..., data: ...} object. file.path is relative to the public/ directory. file.data is the contents of the file. None of the files are written yet. The public array contains not only generated files, but also asset files that should just be copied. It effectively means that the public array contains everything that the public/ folder eventually will contain.

All onCompile plugins are run asynchronously in chain, by using callback. callback takes an error as its only parameter. If a plugin does not need to be run in chain, it could just callback(null) immediately.

Doing something for every file that matches some pattern is now really easy:

for file in public when @pattern.test(file.path)
  do something
callback()

Or asynchronously:

each = require "async-each"
files = public.filter ({path}) => @pattern.test(path)
each files, something, callback

@pattern comes from:

constructor: (@config) ->
  @pattern = @config.plugins.myPlugin?.pattern ? /\.myExt$/

If you want to modify a file, you just modify file.data:

file.data += "/* Easter egg at EOF! */"

If you want to create a new file, just append it to the public array:

public.push path: file.path + ".gz", data: compress file.data

When brunch has run all onCompile plugins it finally writes whatever is in the public array to the disk.

Example

If something like this comes true, I'd like to write a hash-filename-like plugin:

path   = require "path"
crypto = require "crypto"

module.exports = class Hasher
    brunchPlugin: yes

    constructor: (@config)->
        @options =
            pattern: /\.(js|css|jpg|png)$/
            algorithm: "sha1"
            length: 11
            mapPath: "map.json"
        for option, value of @config.plugins.hasher ? {}
            @options[option] = value
        return

    onCompile: (public, callback)->
        map = {}

        for file in public when @options.pattern.test(file.path)
            newFilePath = insertAfterFilename(file.path, hash(file.data, @options))
            public.push({path: newFilePath, data: file.data})
            map[path.basename(file.path)] = path.basename(newFilePath)

        public.push({path: @options.mapPath, data: JSON.stringify(map, null, 2)})

        callback(null)

hash = (string, options)->
    crypto.createHash(options.algorithm)
        .update(string)
        .digest("hex")
        .substr(0, options.length)

filenameRegex = ///
    ^\.?  # A filename is allowed to start with a dot,
    [^.]* # but then cannot contain dots.
    ///

insertAfterFilename = (file, string)->
    path.join(
        path.dirname(file),
        path.basename(file).replace(filenameRegex, "$&-#{string}")
        )

Doesn't that look great? The "map.json" file produced could then be fed into a function used in some templating language. static-underscore-brunch could be used to update all paths in static HTML etc.

Notes

I've been looking through the plugin list, and my conclusion is that lots of plugins will benefit from this change. It's backwards incompatible though. But I'd be happy to contribute to plugins to use the new API. And while speaking about the API: I love how simple it is, don't you think?

[es128]

I haven't fully absorbed this proposal yet, but just wanted to note that it does not have to be backward incompatible. The old/new method signature can be detected, as we've done before when evolving other plugin methods.

[lydell]

Ah, of course, that's great.

Well, the main thing here is I want to avoid reading and writing to the filesystem all the time and proper chaining. All the cache busting, hashing and gzipping stuff are just examples and use cases.

Oh, and while I'm at it: The files of the public array should probably have a .sourcemap property too.

[lydell]

Hmm, does autoreload-brunch depend on that the files are already written to disk when it is run? after-brunch probably does. But all other plugins would benefit. So do we need onAfter or something?

Btw, I could try to implement this, if it is accepted.

[lydell]

Does it hurt performance to read+write instead of copying assets?

[paulmillr]

Yes, we need another API, but onAfter doesn't sound great!

Does it hurt performance to read+write instead of copying assets?

yes, but I guess not much

[es128]

Why another one? If onCompile gets an optional callback to support chaining where necessary, then what use case is not covered? Some current implementations of onCompile may be better off using the optimizer API.

Hmm, does autoreload-brunch depend on that the files are already written to disk when it is run?

Doesn't onCompile already wait for the files to be written before it is run? It should. If a plugin wants to do something before that, it should be a compiler or optimizer.

Any older onCompile implementations that do not accept the callback as an argument will be assumed to be able to run asynchronously - as if the callback was invoked immediately.

The current semantics of optimizer probably should be examined, because in the brunch work flow it really means any work you want to do on the post-compiled and concatenated output files of a specific type (js or css) before they are written. That may be something other than minification or the like. Using a more general term (postCompiler?) may make this clearer to plugin authors.

[lydell]

Yes, onCompile is run after files are written, currently. But I proposed that it should be run before they are, to avoid lots of extra file reading and writing.

It doesn't really matter to me what we improve in the end. I just feel like lots of current plugins could get more help from brunch. I also want to develop that hash-filename plugin, but it's no fun currently.

Using optimizer instead of onCompile - why not. Just remember that many plugins work with any type of file (js, css, html, others).

[es128]

Yeah I think it'll be hard to pipeline something the manipulates filenames - that should probably be an onCompile that happens after the original file is written. Very little perf benefit to jumping through hoops to allow it to happen prior to brunch's regular file write. The use cases are so varied, I don't think it's worth trying to standardize this and narrow the use-cases it can be used with. onCompile is perfect for letting a plugin do anything it wants to do after brunch has run its pipeline.

A single plugin can potentially do both - an optimize method to manipulate paths/refs within compiled source code and an onCompile hook to actually rename files.

[lydell]

We just must remember that a file created by one plugin should be readable by another. For example, my "hash-filename" plugin creates app-HASH.js, then I want it to be gzipped by another plugin. That could be solved by letting the plugins run in chain via a callback. Still, reading files is boring ...

[lydell]

Perhaps a bit can be solved by more documentation. For example, what's the proper way to read the public folder?
readDirRecursive(@config.paths.public)?

[es128]

Ok.. I hadn't really absorbed what your proposed public array was about until now. I would think output is a better name. I understand the value of this concept better now, and am now of the opinion that it probably does need a whole new API in order to be done effectively. So the overall pipeline would be like lint -> compile -> concat (internal) -> optimize -> postProcess (let's think of a good name) -> write to fs -> onCompile.

+1 on this concept from me if you want to go ahead and work on it (assuming @paulmillr is on board).

[paulmillr]

Why optimize -> postProcess and not reverse?

[es128]

optimize remains a per-type final intervention as it is now. postProcess is for more general intervention right before the files are going to be written, including the ability to rename, etc. The gzip use case, for example, wouldn't work if it was done before optimize.

[paulmillr]

So it receives all files?

[es128]

Yes, look at the first paragraph of @lydell's "Proposal" above. It receives the path and data of everything brunch was about to write to the file system, and allows the plugin to mutate it before it's written.

[paulmillr]

Yeah great idea.

[lydell]

I thought that I could start working on this the past days, and continue this weekend, but things came in between, unfortunately :(

[es128]

let me know when you start - I may end up working on it in the meantime if you don't

[lydell]

Will do, but don't expect anything at least before Christmas.

[mutewinter]

I would love to see this API be added as well. I just finished digest-brunch, which implements the example above, but feels like a hack.

[lydell]

I still haven't started working on this, and it looks like it won't happen soon :(

[roperzh]

Hello everyone!

I'm facing the same issue trying to implement a plugin. Do we still want to implement this? If the answer is yes, I can take some time during the week to send a PR.