How to access masked data?

[jgonggrijp]

This feature/use case was previously discussed at length in #11, which has many links to other discussions as well as to implementations of partial solutions.

I create a new ticket for it now, so I can refer to this when discussing power lambdas in the nearby future.

In short, with these YAML input data (copied with correction from @pvande's opening post in #11):

name: 'Hello World'
users:
  - name: 'Bill'
    pets:
      - name: 'Yoda'
        type: 'Cat'
      - name: 'Ada'
        type: 'Rabbit'
  - name: 'Steve'
    pets:
      - name: 'Wayne'
        type: 'Llama'

There is currently no way in standard Mustache to write a template that will give the following output:

Bill owns a Cat named Yoda.
Bill owns a Rabbit named Ada.
Steve owns a Llama named Wayne.

The following template would fail, because the name of the pet masks the name of the user (owner):

{{#users}}
{{#pets}}
{{name}} owns a {{type}} named {{name}}.
{{/pets}}
{{/users}}

Handlebars would address this by writing {{../name}} to access the name of the owner. It also has the special keyword @root, which would make it possible to obtain Hello World on every line by writing {{@root.name}}. However, there seems to be universal agreement among Mustache implementers that ../ is too ugly and that special keywords like @root should be avoided. I personally share these opinions, too.

I have seen a number of (partial) solutions suggested. Below, I go over each. None of them seems really ideal to me, so I hope that someone will come up with a better solution, or convince me that one of the solutions is good enough.

Prevent masking from happening

In general, the input data should be prepared before feeding them into the template. In the above example, one could argue that the name of a user should be called userName instead and that each pet should have a petName. The template becomes trivial in that case:

{{#users}}
{{#pets}}
{{userName}} owns a {{petType}} named {{petName}}.
{{/pets}}
{{/users}}

Purists who want to keep the syntax as simple as possible, will argue that this is the only acceptable solution. End users might argue that renaming keys in this way is not always possible and desirable.

Anchored stack descent syntax

Basically, this amounts to some kind of equivalent to Handlebars's ../ notation. I saw @determin1st suggest a single dot per level in #135 (comment). In that case, the template would look like this:

{{#users}}
{{#pets}}
{{..name}} owns a {{type}} named {{name}}.
{{/pets}}
{{/users}}

We could also write {{.name}} for the name of the pet. This would have a slightly different meaning than just {{name}}: if name is not found in the current context, lookup does not fall through to lower levels of the context stack. Hence "anchor": the dots before the name set the level to start looking for the name, but this is also the level to stop looking.

@bobthecow implemented a subset of this variant in Mustache.php, where you can use .name to anchor resolution to the top of the context stack, but not prepend additional dots to descend into the stack.

While anchored descent makes it possible to reach all versions of a key in principle, this crucially depends on the template author having a lot of knowledge about the contents of the context stack. If a key is sometimes present at a particular level but not always, or if the template may or may not have recursed through a partial, it becomes easy to miss the key at one context level. This might happen, even though there is a viable fallback value at a lower level.

A related problem is that the bottom of the context stack cannot be reliably addressed.

With respect to syntax, this extension is relatively safe, as the spec currently does not recognize names that start with a dot, unless the dot is the entire name.

Root anchoring

This amounts to giving a name to the context root, so that dotted names can start with it. This is Handlebars's @root. I saw @pvande suggest @ as a shorter name in #11. Hello World could then be obtained with {{@.name}}.

The main problem with giving any name to the context root, is that it potentially breaks existing templates if the name already happens to be used as a key. Either, users can no longer use the name as a key, or they can, but then we are back at the masking problem.

Root anchoring is less powerful than descent anchoring, because items inside lists do not have an obvious path relative to the root. Without additional changes to the language, we still cannot access the name of the current user inside the {{#pets}}{{/pets}} section. Below, I have indicated the problem with a question mark:

{{#users}}
{{#pets}}
{{@.users.?.name}} owns a {{type}} named {{name}}.
{{/pets}}
{{/users}}

To make this work, we would either need to agree on something that can be written instead of the question mark, or invent a way to name the current iteration over a list so we can use this instead of the entire @.users.? prefix.

At first sight, it might seem to make sense to write . instead of the ?, because the dot already stands for the implicit iterator. However, this introduces multiple dots in the middle of a name, which do not all mean the same. This can easily confuse users, especially if you do it multiple times in a row. Moreover, it is a breaking change for implementations, since multiple dots in a row currently means nested empty keys, which many implementations probably do not allow.

One may then suggest to use something other than the dot, for example the wildcard *. However, this has the same problem as the root anchor itself: it might conflict with existing keys and will then either break templates or reintroduce the masking problem.

Both ways to fill the ? also suffer from the problem that such notation complicates the context lookup algorithm.

A way to name the current iteration over a list could be obtained using filters (#153). For example, if we imagine an enumerate filter that maps every list element to an {index, value} pair, the following template would work:

{{#users | enumerate}}
{{#value.pets}}
{{value.name}} owns a {{type}} named {{name}}.
{{/value.pets}}
{{/users | enumerate}}

This is only a very partial solution; if we also enumerate the pets, value itself if masked and the template will fail again. This solution is also problematic because filters are currently nonstandard.

Prioritized keys

This is a partial alternative to root anchoring that @groue implemented in GRMustache (docs). It does not use special keywords and therefore avoids the naming problems of the previous solution. Instead, users can use the GRMustache API to mark particular keys in the context root as prioritized. Those keys cannot be masked by higher context frames.

This solution cannot be used to make our example work, but I mention it for completeness, since it is still a way to disambiguate context keys. It should be thought of as a security feature rather than as a way to make working with the same key at multiple context levels convenient.

Skips

Skips are an alternative to anchored descent, where you do not restrict the lookup to a single context level, but instead tell the engine to continue looking after first finding a match in the context. I saw this first described by @pvande in #11, where he suggested adding a suffix to the name with one quote ' per skip. With this notation, our template would look like this:

{{#users}}
{{#pets}}
{{name'}} owns a {{type}} named {{name}}.
{{/pets}}
{{/users}}

We could also add Hello World to each line by adding {{name''}}.

The number of skip quotes will always be less than the number of dots with anchored descent. There are two reasons for this: firstly, because the top level does not count, and secondly, because skips only count context levels where the key is actually present.

Besides the fact that fewer quotes are needed, skips are safer than anchored descent in two important ways. Firstly, they require far less knowledge from the user. The Mustache implementation simply retrieves each key from wherever it happens to find it, except that it now continues looking for a second or third occurrence. Secondly, if there is a fallback value, the skip notation will always find it. The user cannot miss it by selecting the wrong level (though he or she can still miss all candidates by making too many skips).

Sometimes, a user might want to miss if a key is not present at a particular level, for example the top. Skips cannot do this, and in this sense, they are less powerful than anchored descent. They also still suffer from some of the same problems as anchored descent: the context root cannot be reliably addressed, and the user might miscount the number of skips when optional values or recursion are involved.

The suffix quote notation is problematic because it may conflict with existing keys. This could be addressed by using dots instead, either as a suffix or a prefix, but this again has the problem of being potentially confusing with the other meanings of the dot.

Lambdas or filters on steroids

Instead of adding new syntax, we could extend the semantics of lambdas (or filters) so that they can retrieve data at lower context frames. I suggested this in #135, as follows:

Input data with lambda (JavaScript)

{
    parent: function(content, magic) {
        return magic.getContextFrameFromTop(1);
    },
    name: 'Hello World',
    users: [ /* as before */ ]
}

template

{{#users}}
{{#pets}}
{{parent.name}} owns a {{type}} named {{name}}.
{{/pets}}
{{/users}}

We could also, for example, implement a root lambda with magic.getContextFrameFromBottom(0). The latter would allow us to get Hello World with {{root.name}}.

This would achieve the same expressiveness as in Handlebars, without any of the syntactic problems. The main downside is that it requires adding a capability to lambdas (or filters), which maybe does not belong there. The convenience and safety of skips could not be matched in this way, unless lambdas also gain a capability to perform context lookups at will.

Edit: I later figured out a way to elegantly implement skips using supercharged filters, which would also be able to reliably obtain the bottom-most fallback value. See #154 (comment). In addition, I corrected the mistakes that @determin1st pointed out in #154 (comment).

[determin1st]

@jgonggrijp

Basically, this amounts to some kind of equivalent to Handlebars's ../ notation. I saw @determin1st suggest a single dot per level in #135 (comment). In that case, the template would look like this:

{{#users}}
{{#pets}}
{{...name}} owns a {{type}} named {{name}}.
{{/pets}}
{{/users}}

nope, it will look like this:

{{#users}}
  {{#pets}}
    {{..name}} owns a {{.type}} named {{..pets.name}}.
  {{/pets}}
{{/users}}

...the dot already stands for the implicit iterator.

ill repeat that parts of mustache grammar/definitions are nothing but distorted meanings. this one is called "iterator" because originally it was related to iteration. after upgrading my own implementation to support . i named this feature as explicit stack selector, or dot selector. so, stack selector is a path modifier which can appear only as prefix to the path.

i belive that . selector with positive backpedal (multiple dots .. ... .... etc) to address previous stack values is rare use.
inside of my lambdas it will look like:

{
    lambda: function(m) {
        return m.value("..pets.name");
    },
}

other suggestions are doable through . or naming, so i dont see any viable cases for the rest.

[determin1st]

if its a list, sure, the ..pets.name wont work. three dots wont work either:

but two dots will do intended job

[jgonggrijp]

You skipped the square brackets of the pets list.

[determin1st]

pets dont go into stack, pets[n] does

[jgonggrijp]

Ah, you're right. I will correct it!

[determin1st]

users also wont go, its data.* instead of users

[jgonggrijp]

I just came up with a way to supercharge filters (#153) so they could implement skips as well as reliable bottom addressing, that doesn't seem wildly out of place and which should also be straightforward to implement and use.

In a filter chain, the first filter, here parent,

{{ name | parent }} owns a {{type}} named {{name}}.

receives two arguments, if it can handle it. The first argument is just the regular value that it would always receive and that would be resolved from the context based on the key left of the pipe, here name. If we are currently at the first pet of the first user, that value will be "Yoda". The second argument is a list with any fallback values that were found deeper in the context stack, deepest last. In this case, that list will be ["Bill", "Hello World"].

With this semantic, the parent filter, which performs a single-level skip, is trivial to implement. It looks like this in JavaScript:

{
    parent: function(value, fallbacks) {
        if (fallbacks.length < 1) return undefined;
        return fallbacks[0];
    }
}

A bottom filter, which does something similar to Handlebars's @root, is also trivial to implement:

{
    bottom: function(value, fallbacks) {
        if (fallbacks.length < 1) return value;
        return fallbacks[fallbacks.length - 1];
    }
}

The value retrieved in this way is not necessarily at the root frame of the context, but it is guaranteed to be the bottom-most value with the given key in the context stack.

There are some details to discuss about the implementation of this idea; I will take those to #153. In any case, this is currently my favorite way to address masking. What do others think?

[jgonggrijp]

Technical discussion here: #153 (comment).

[spullara]

I vote for "Prevent masking from happening" as usual. If there was no other option in Mustache except to make the input data or backing code do the right thing it would be easier to explain how to use mustache. I would point out the example data makes it possible to render what you asked for without changing the spec so a better motivating example would be good:

{{#users}}
{{name}} owns a {{#pets}}{{type}} named {{name}}{{/pets}}.
{{/users}}

[jgonggrijp]

If there was no other option in Mustache except to make the input data or backing code do the right thing it would be easier to explain why mustache.

Could you phrase this in a different way? English is not my native language and I cannot make sense of this sentence.

The intention of the example template is to generate a separate line that repeats the owner's name for each pet. The alternative template that you suggested will not do this. I will give Bill a second pet to clarify this.

[spullara]

It is much easier to explain the correct way to do something in Mustache if it remains very simple and almost everything interesting is in the backing code / data is another way of putting it.

As for the example, changing it so they have multiple pets will help though it still doesn't feel like a real example to me as you are unlikely to want to repeat the users name for each pet. Having strong examples to motivate features like the ones we have for inheritance I think would help.

[jgonggrijp]

Thanks for clarifying that.

If we replace users by classes and pets by methods, and instead of a sentence about ownership, we render a documentation paragraph in which we concatenate the names as class.method, does that make the example more convincing to you?