Power lambdas (suggestions for a better name welcome)

[jgonggrijp]

This is a continuation of the discussion in #135. Before that, there was also relevant discussion in #19 and #41, including a comment that hinted at something like power lambdas in #41 (comment).

At the time of writing, (regular) lambdas have the following required capabilities, according to the interpolation and sections specs:

• If used in a section, receive the template content of the section as a string argument.
• Return a value that is rendered (for interpolation) or used as the new context (for sections), instead of the lambda (function/method) itself.

The optional lambdas extension adds one more capability:

• If the return value of the lambda is a string, render it as a template. In a section, this template replaces the original content of the section.

Power lambdas would supercharge the capabilities of lambdas, while changing as little about the language as possible. There is no syntactic change and only one semantic change: lambdas that can handle it receive an additional argument, which I refer to as the "magic".

The magic adds one or more of the following features to the existing capabilities of lambdas, in an implementation-defined way. Implementations are free to omit some of them, but encouraged to support as many as they can. However, it is up to debate whether all of the following features should be included in the spec. More on this below.

Edit: I no longer believe features 1-3 are needed, see #155 (comment).

1. Retrieve a list of all keys that are visible in the context stack.
2. Look up the value of a given key in the context stack. Ideally, it must also be possible to see which context frame underlies the value that was found.
3. Retrieve any frame in the context stack (intended as read only).
4. Override the content with an external template that was already compiled beforehand (rather than returning a string that will be compiled or interpreted as a template on the fly).
5. Override the content with a final string that is not processed further (rather than returning a string that will be compiled or interpreted as a template). For sections, this string will not be escaped; if the string needs to be escaped, the power lambda should do it.
6. Render a template of choice against the current context stack and obtain the result as a string, without it necessarily being interpolated.
7. Define a prefix and/or suffix, in order to wrap whatever ends up being rendered in the place of the interpolation tag or section. This can currently already be done in section lambdas by concatenating the prefix, section content argument and prefix and returning the resulting string as a new template, but this feature makes it possible for interpolation tags as well and removes the necessity to compile or interpret a template at runtime. Implementations may support static strings, template strings or precompiled templates as prefix/suffix, or at their option, multiple of those.

I envision this as an optional extension module, perhaps ~power-lambdas.yml. It would follow a slightly different format from the existing ~lambdas.yml: each implementation that attempts to pass the specs in that module will need to bring its own lambdas.

The first three features mostly cater to a lambda's ability to return a value that gets pushed on the context stack. The other four mostly cater to a lambda's ability to override the template inside a section. There is some interaction between these sets as well. In detail, the features were originally motivated as follows:

• 1+2 enable advanced masking. For example, a lambda might return a new context frame in which every list in the context is masked by (a lambda proxy to) an enumerated version of that list. Such masking would make lambdas capable of everything that filters could do (see Filters (or pipes) #153) and more. Where a filter would be written {{# someList | enumerate }}, the equivalent power lambda would be written {{# enumerate.someList }}.
• 2 alone would suffice to implement a more limited alternative to filters that only works for interpolation. Instead of {{ someString | capitalize }}, one would be able to write {{#capitalize}}someString{{/capitalize}}. However, passing section content as argument text to lambdas in this way would also open some options that filters do not have, since the lambda is free to interpret the section content in any way that suits the application. Feature 5 could make this use case more efficient.
• 1+2+3 OR 1+(an enhanced version of 2) would make it possible to implement skips (see How to access masked data? #154) without adding any new syntax to the Mustache language.
• 3 alone would make it possible to implement anchored descent as well as root anchoring, as discussed in How to access masked data? #154.
• 2+4 OR 2+5+6 would enable an efficient lambda-based implementation of dynamic partials. Instead of {{>*animal}} one would write {{#dynPartial}}animal{{/dynPartial}} (for example). 2 alone would also suffice, but would require recompiling the external template every time. This use case is no longer relevant because dynamic names already made it into the spec in the meanwhile.
• 4 OR 5+6 OR 7 would make it possible for a lambda to override or extend a local template, without that template having to be compiled at runtime. This would be helpful for implementations that are safely typed, such as JStachio by @agentgt.
• 4 OR 7 would also make it possible for a lambda to override or extend a local template and push a new frame on the context stack at the same time.
• 5 alone would restore a capability that was lost with the lambdas extension: to return a string that will be interpolated verbatim (or used as context frame), without it being subject to expensive template processing.
• 5+6 would enable an inversion of control, where the lambda renders a template and returns a result that is interpolated, instead of it returning a template that is then rendered and interpolated. Together with the existing capability to read the content of a section, this would enable me to write a Mustache equivalent of handlebars-i18next.
• More generally, 6 alone would enable lambdas to derive information from the context. In this sense, it could also be considered a weak alternative to feature 2.
• 7 alone would provide an optimization for a use case that is often seen in example code: take the content of a section, add a prefix or suffix to it and return the result as a template override.

For the first two motivations, where power lambdas provide a close alternative to filters, I now believe that filters are more appropriate, even though power lambdas would be more powerful. Filters are much easier to implement for a given application: a filter can be any straightforward function in the host language that maps one value to another, while a power lambda requires specific knowledge about the Mustache engine as well as intricate masking and/or argument parsing logic. In addition, I cannot think of a convincing situation where a filter-like use case really requires the additional context awareness or argument parsing that a power lambda could provide.

The next two motivations, which position power lambdas as a solution for the masking problem, are more convincing in my opinion. Syntactic alternatives are discussed in #154, but I feel none of those alternatives is very good, so I currently still have a slight preference for the lambda approach. That said, if a syntactic (or other non-lambda) solution for the masking problem does make it into the spec, this completely removes any justification for feature 3. Update: I think I have now come up with a very good alternative, see #155 (comment). Features 1-3 can all be left out if that alternative is implemented.

Dynamic names are in the spec already. I personally find the remaining motivations very convincing, but since those only involve features 4-7, this makes the inclusion of features 1-3 a bit uncertain as I see it. Of the latter three, 2 is probably the most useful one, but also the most redundant with filters. Still, there might be other good arguments why 1-3 need to be included anyway.

7 is nice to have, but not strictly necessary if 4-6 are also available. Thus, 4-6 seem to be the most "core" features of power lambdas, with 7 being a welcome extra.

An interesting option would be to return a lambda from a filter. That is, a filter might construct and return a closure, bound method or partially applied function with knowledge about a value in the context. That function could then proceed to act as a lambda, with context argument parsing, template overrides and/or pushing a new stack frame. I have discussed this option in #153 in more detail. It could provide a partial alternative to features 1-3, though in many cases, it may not be enough to address the masking problem.

[jgonggrijp]

In #135, @agentgt suggested investigating exising implementations of "more powerful" lambdas, in order to get an overview of existing practice. I replied with this list of implementations I was already aware of:

• Near the beginning of Idea: power lambdas #135, @bobthecow described the power features in Mustache.php. If I understood correctly, he implemented at least feature 6 (render an arbitrary template against the current context stack and obtain the result as a string), but not feature 1 or 3 (full read access to the context stack).
• Mustache.php also implements filters (Filters (or pipes) #153), though I would consider that just "different" from lambdas rather than "more advanced". I mention it for completeness.
• In Proposition for a mechanism to add header/footer to sections. #147 (comment), @spullara mentioned having "more powerful lambdas" in Mustache.java. He didn't clarify what exactly makes them more powerful, but I suppose that could be found in the documentation.

[jgonggrijp]

In #147 (comment), @agentgt briefly returned to this topic with the following comment:

(...) I believe we were talking about searching for other Mustache implementations that have power(like) lambda support and JMustache does and JStachio was largely modeled after it with the exception of getting the whole stack (JStachio does only top of stack): https://www.javadoc.io/doc/com.samskivert/jmustache/latest/com/samskivert/mustache/Mustache.Lambda.html

We have been using a forked version of JMustache for almost a decade. It's small yet fairly powerful. In fact we tried switching to Mustache.java several times but found it too hard to lock down for our needs (we have quasi users editing templates) and I could not port some lambdas that we had as Mustache.java does not have stack access last time I checked. (...)

[jgonggrijp]

In #153 (comment), I came up with a flavor of "power filter" that could address the masking problem. I think it provides a cleaner alternative to features 1-3. It is less powerful, but still powerful enough to make me want to leave features 1-3 out of the power lambdas specification!

[agentgt]

@jgonggrijp On discovery of other implementations: Another Mustache variant in Java land that has sort-of power lambdas is Trimou: https://github.com/trimou/trimou

Trimou went the Handlebars helper route. I'm not sure why Trimou never got that popular. My guess is people in Java land would either just accept the decorating or prebuilding of models (i18n) required in Mustache.java (@spullara) or use actual Handlebars.java. Or probably just marketing and network effect.

Several times I have considered experimenting with a Handlebars helpers like extension to JStachio but the only real use case I seem to need it for was i18n.

Sort of an open question but I have to wonder how much of power lambda desire / use case is i18n particularly parameters?

In the past when we did i18n in my company we used sort of a giant key/value (Map<String,String>) with multiple overlays (e.g. first try this map then try the other). There were some scaling issues but generally worked particularly if we used sessions.

In other cases the model would just get all of the i18n resolved apriori.

I guess it would be interesting to me and possibly helpful to others if we had a list of fairly concrete use cases power lambdas could solve.

Otherwise it appears where we have been using JStachio/JMustache lambdas a filter could have been used instead (well I'm not so sure on the whole fallback stuff but the transform this node to another node is mainly what we are using lambdas for).

[spullara]

That would be the part where it does a lookup in the translation hash map of the translation key as a stand in for some real translation system with a fallback to the default text provided. As a bonus this also demonstrates having a global context that does the I18N work.

[agentgt]

I guess I'm just not sure what happens here given that payload is an object:

var translation = i18next.t(this.keys, content, payload);

The payload in the above is the friend object (unlikely to be a key). Maybe it was just a typo: https://www.i18next.com/overview/api#t

[jgonggrijp]

@spullara your example seems to rest on the assumption that i18next.t(['goodmorning'], 'Good morning, {{name}}', {name: 'Sam'}) is supposed to return '좋은 아침이에요 {{name}}'. It is not; it returns '좋은 아침이에요 Sam'. Also, your t lambda is not accessing the payload from my example, and I see no way in which it could. @agentgt recognized that correctly. Nice touch with the Korean, by the way.

@agentgt The i18next documentation is admittedly less than crystal clear. The payload argument is explained in these documentation sections:

• https://www.i18next.com/translation-function/essentials#overview-options (see replace)
• https://www.i18next.com/translation-function/interpolation#basic

[spullara]

Gotcha. Didn't realize that i18next is also a templating language that differs enough from mustache you can't figure out what needs to be made available. So are you planning on passing the entire context as JSON? That isn't possible in mustache.java since what fields that are available aren't enumerable. If it was enumerable it would be pretty easy to support it, but it is instead impossible unless you limit what is allowed in the context to Maps or JSON or some other pure data objects. IF you limited i18next content to compatible mustache templates I could build it as well OR if there was a way to ask i18next what keys are accessed by the content.

Mixing templating languages like this seems like a bad idea.

[jgonggrijp]

Right, so these are my two points:

1. Mixing templating languages is one of the things I need in this case, and I can do it in a safe way by the control inversion that would be unlocked by power lambdas (liberated flavor).
2. Retrieving arbitrary information from the context stack is the other thing I need. In my example, I used a filter to do it, but the introspective flavor of power lambdas would also do the job.

I don't need to pass the entire stack to i18next, I can just select a single frame because I know the default string. i18next is not really a templating language. It only does interpolation, but it does it in a localization-aware way that I think would be beyond the scope of Mustache.