feat: add support to per-function configuration files

[eduardoboucas]

BEGIN_COMMIT_OVERRIDE
feat: add support to per-function configuration files (#1030)

feat: new bundler that allows to skip bundling (#1173)
END_COMMIT_OVERRIDE

Summary

Adds support to per-function configuration files. These files are only taken into account if they are inside a directory that is part of the new configFileDirectories configuration property.

Take the following project structure as an example:

.netlify/
  functions-internal/
    internal-function-1/
      internal-function-1.json
      internal-function-1.js
    internal-function-2.json
    internal-function-2.ts
netlify/
  functions/
    user-function-1.ts

If configFileDirectories contains .netlify/functions-internal, then internal-function-1 and internal-function-2 will read configuration properties from .netlify/functions-internal/internal-function-1/internal-function-1.json and .netlify/functions-internal/internal-function-2.json, respectively.

Even if there was a netlify/functions/user-function-1.json file, it would be ignored if netlify/functions was not part of configFileDirectories.

Any configuration properties loaded from a per-function configuration file take precedence over the global config object.

A picture of a cute animal (not mandatory, but encouraged)

[github-actions]

⏱ Benchmark results

Comparing with b472a84

largeDepsEsbuild: 7.5s

⬆️ 4.15% increase vs. b472a84

^                                   7.5s  
│                           7.1s    ┌──┐  
│   6.6s                    ┌──┐    |▒▒|  
│ ──┌──┐────────────────────┼──┼────|▒▒|──
│   |  |    6.1s     6s     |  |    |▒▒|  
│   |  |    ┌──┐    ┌──┐    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
└───┴──┴────┴──┴────┴──┴────┴──┴────┴──┴──>
    T-4     T-3     T-2     T-1      T    

Legend

• T-4 (344f4ae): 6.6s
• T-3 (e1200a2): 6.1s
• T-2 (850404d): 6s
• T-1 (b472a84): 7.1s
• T (current commit): 7.5s

largeDepsNft: 36.9s

⬆️ 3.83% increase vs. b472a84

^                                  36.9s  
│                          35.5s    ┌──┐  
│                           ┌──┐    |▒▒|  
│ ─30.7s────────────────────┼──┼────|▒▒|──
│   ┌──┐                    |  |    |▒▒|  
│   |  |           27.6s    |  |    |▒▒|  
│   |  |            ┌──┐    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |            |  |    |  |    |▒▒|  
│   |  |    n/a     |  |    |  |    |▒▒|  
└───┴──┴────────────┴──┴────┴──┴────┴──┴──>
    T-4     T-3     T-2     T-1      T    

Legend

• T-4 (344f4ae): 30.7s
• T-3 (e1200a2): n/a
• T-2 (850404d): 27.6s
• T-1 (b472a84): 35.5s
• T (current commit): 36.9s

largeDepsZisi: 55.3s

⬆️ 3.96% increase vs. b472a84

^                                  55.3s  
│                          53.1s    ┌──┐  
│                           ┌──┐    |▒▒|  
│                           |  |    |▒▒|  
│ ─45.5s───45.4s────────────┼──┼────|▒▒|──
│   ┌──┐    ┌──┐   42.5s    |  |    |▒▒|  
│   |  |    |  |    ┌──┐    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |▒▒|  
└───┴──┴────┴──┴────┴──┴────┴──┴────┴──┴──>
    T-4     T-3     T-2     T-1      T    

Legend

• T-4 (344f4ae): 45.5s
• T-3 (e1200a2): 45.4s
• T-2 (850404d): 42.5s
• T-1 (b472a84): 53.1s
• T (current commit): 55.3s

[danez]

@eduardoboucas What was the idea behind restricting this to certain directories instead of allowing it for every function? We did not want to expose this config support to user-generated functions?

Also do you think that zisi is the right place for reading the config? The problem I have with it is that not all of the config is relevant for zisi, lambda_runtime being one example. That means that zisi now reads a config file that might only contain configuration for a later step outside of zisi.
Wouldn't it make sense to have the logic for the function config files to be inside @netlify/config? And then zisi gets whatever config it needs from there. Same actually for ISC and scheduled functions. Might make sense to move that to @netlify/config as well, because in the end it is configuration that can be supplied to zisi.

[eduardoboucas]

What was the idea behind restricting this to certain directories instead of allowing it for every function? We did not want to expose this config support to user-generated functions?

Yes, exactly! Once we get to ISC, I think we'll want to allow all types of functions to be configured that way, but until then I don't think it makes sense for user-generated functions to use this functionality. Asking people to write a per-function JSON feels incredibly unergonomic, and they'd be better off using netlify.toml instead.

Also do you think that zisi is the right place for reading the config?

Note that zip-it-and-ship-it is the only package capable of resolving a functions config. Imagine this:

[functions."next_*"]
node_bundler = "nft"

@netlify/config doesn't know how to unpack this into a list of functions, because it doesn't know how to tell whether a file is a function. This is why zip-it-and-ship-it is currently responsible for handling functions configuration.

Personally, I don't think that passing to zip-it-and-ship-it a configuration property that it doesn't use or understand is that big of a problem, but I'm happy to discuss any alternative options you have in mind.

[danez]

Other question: The Proposal only specifies nodeBundler and included_files as valid configuration options coming from the deployment config. The current implementation does not limit the config though, that means all config options from zisi are currently available to be set in the config.

Should we limit/restrict this?

I personally don't see any downside of allowing all options, but I probably am missing something.

Should we add validation of the config? We can also do that later, once we introduce non zisi config stuff. At least then it would make sense IMHO.

@jackiewmacharia @eduardoboucas

[eduardoboucas]

I also don't see any obvious downsides of allowing all configuration properties. I think the proposal mentioned those two as examples of properties we have an identified use case for in the deploy configuration API.

[jackiewmacharia]

Looks good overall, left a few comments.

[danez]

@eduardoboucas I cannot add you as reviewer, because you are the creator. would you mind looking over basically your own code once more, and comment if you notice something?

[eduardoboucas]

would you mind looking over basically your own code once more, and comment if you notice something?

Done!

[eduardoboucas]

LGTM!

[danez]

Okay I consider this done. I'm not going to merge this yet, but will start to build the skip bundling based on this PR and then merge both together.