Tera v2 wishlist/changes

[Keats]

On top of https://github.com/Keats/tera/issues?q=is%3Aopen+is%3Aissue+label%3A%22For+next+major+version%22

Parser

Operator precedence and expressions ✅

Parentheses should work everywhere and precedence should make sense. This is mostly already implemented in a new parser (private for now).

Should something like {{ (get_page(path=“some-page”)).title }} work? Right now it requires going through set which is probably fine imo.

Better error messages ✅

The new parser is hand-written so we can provide detailed error for each type of error. I’m also thinking of spanning all expressions in the AST for better rendering errors but not 100% convinced on that yet, will need to try.

Whitespace management

Do trim_blocks and lstrip_blocks (https://jinja.palletsprojects.com/en/3.0.x/templates/#whitespace-control) by default that can be switched on/off as one thing and not two.

Indexing changes ❔

Right now you can access array indices by doing my_value.0 or my_value[0]. This is in line with Jinja2/Django but it feels a bit weird.

How about allowing only my_value[0] so array index is the same as dict index and more like all programming languages (except tuple access)? The dot syntax doesn’t work when the index is a variable anyway which is probably the main usage of indexing. It would be a big departure from Django/Jinja2 though.

Features

call

https://ttl255.com/jinja2-tutorial-part-5-macros/#call-block is a good example of that feature.
I do find the naming in Jinja2 very confusing though so it would probably be called something different

Investigating valuable ❔

https://tokio.rs/blog/2021-05-valuable
This should avoid an awful lot of cloning and would improve performances a lot. Still not clear whether it will work though.

Improving filters/tests/functions

They should be able to get values from the context implicitely the same way expression do so we don’t have to repeat things like lang=lang in Zola for example.
Maybe a first argument which is something like get_ctx: |key: &str| -> Option<Value>?
It would be nice if there was a way to define the arguments in a better way than the current macros too.
Related issue: #543

Also remove some weird things added to match Jinja2 like #650 (comment)

--

Any feedback / other items you want to change?

[KillTheMule]

Any feedback / other items you want to change?

I'm not yet a tera user, but I gave it a very serious look, because I want runtime templates (using askama for now, which uses compile-time templates). I'm templating latex files, which use brackets (in particular, the curly ones) all over the place, so what I'd need is different delimiters than {{ etc. Is that something you would consider? It would certainly be feasible to restrict this to 2-char delimiters (askama does this, I'm using ~< instead of {{ right now).

[Keats]

The current parser library doesn't allow passing variables to the lexer and neither does the future lexer library :/
Would adding something [[, [%, [# as equivalent to the curly braces work with Latex?

[KillTheMule]

Unfortunately, latex also uses [ / ] quite a bit as well, I can't imagine that working except for the most simple documents. I'm pondering trying addition pre/postprocessing steps, but that feels pretty brittle.

[Keats]

Does it use [[, [%, [# though? Single [/] would be ignored. Otherwise <<, <%, <# but it seems more common to me than squared brackets

[KillTheMule]

No, [[, [% and [# aren't really used and easily avoided if needed. Is that sufficient, though? Because one could also easily avoid {{ and the like, but other templates libraries if tried (e.g. jinja2) could not deal with the fact that there were single curly braces in the document.

[Keats]

Tera only cares about {{, {% and {# (and their closing equivalents), a single { or } will not be considered. See #642 for a recent example.

[KillTheMule]

Huh, that would be awesome. You mention jinja2 in that issue, but I'm pretty sure it did trip over the brackets in the default settings (but jinja2 allows changing the delimiters, so I got it to work). Nevertheless, I'm taking tera for a test-drive tonight, thanks for your time, and thanks for tera!

[KillTheMule]

After using tera for some time (works nicely, thank you for it!), what I am missing is more fine-grained errors, in particular around rendering a template. When the template has a variable that is missing from the context, the error seems to be a generic Msg type, but I really want to give good feedback to my users (which are very non-technical people, and I cannot assume they speak english), which seems to mean that I have to manually parse the msg string and extract the info that I need. Moreover, if using array variables, it would be great to know which part of the chain was missing (i.e. if using var.field.subfield in a template, and var.field exists, it would be nice to have a possibility to not just show "var.field.subfield missing", but "subfield of var.field missing"... but that's just a shower thought :) A major improvement for me would be to get away from parsing msg strings manually.

[Keats]

So the next parser has spanned expression pointing to the exact span in the template.
Getting which value didn't exists in an expression like var.field.subfield is a bit more annoying but might be doable. That's a good idea.

[MahaloBay]

Hi some ideas for v2 :
I believe that there is currently no possibility of negation (or I misread the doc ^^)
For example, i can {% if my_var %} but not {% if !my_var %}, often I found myself having to do {% if my_var %} {%else%} blablabla {% endif %}, same thing in "is containing", i think negation is a good thing to add

EDIT : i search in doc and i find this

Maybe i was misread and should using not

[Keats]

Yep, you have to use not. So {% if not my_var %} and {% if x is not containing(..) %}

[jsha]

Hi! We're starting to use Tera in rustdoc. I like it a lot so far. One thing we're really interested in is performance. Rustdoc generates a lot of HTML in a single run, and there's often a developer waiting to look at the output, so we care a lot about speed. Also, the rustdoc team has put in a lot of effort speeding it up, so we want to make sure the move to templates doesn't slow things down again.

Right now I'm investigating a perf regression after adding a second template. Some themes that come up:

• malloc and free are responsible for a lot of the regression, suggesting number of allocations might be an issue
• regex is also fairly big, suggesting get_json_pointer could be sped up (Speed up get_json_pointer #678).
• BTreeMap::insert shows up high, suggesting that Context::from_serialize is costly.

The valuable crate looks very useful for reducing allocations, and possibly also for replacing the BTreeMap. My thinking is that when the input is a single struct, tera should be able to directly visit each field of the struct by name without building an intermediate BTreeMap of field names -> values.

[Keats]

We're starting to use Tera in rustdoc.

:o nice!

Performance is the main goal for v2 as well, I'm guessing a lot of allocations you see are from the JSON serialization, which is the bottleneck for Zola as well (minus syntax highlighting). I'm guessing rustdoc is the same (lots of text) that need to be essentially cloned when moving to Value::String.

I have some big hopes for valuable to reduce/remove those allocations but it seems the project stalled a bit.

[jsha]

Has it stalled? Looks like they're planning an initial release as of 8 days ago.

[Keats]

I am looking at tokio-rs/valuable#59 which would be required for Tera

[KillTheMule]

One thing I'm wishing for right now was a way to get a template from tera to inspect it. In my case, I can probably just read the file from disc again, but depending on the structure of the program it might be worthwhile to provide something like that (never mind that it might just be faster than reading it again). Maybe a use case would be modifying those on the fly? Or check for certain content? The latter is what I want to do btw.

[Keats]

You can access a template AST, it's just not visible in the documentation since the AST is not stable.

[Keats]

It would be nice to have context local functions, it was pulled from v1 due to some breaking changes but it's on the table for v2

[ssendev]

I would like to have something like Components from Vue.js which in Tera would probably be macros 2.0.
The difference to current macros being that they not only take arguments but also can have slots blocks.

{% macro user_list(users, user_icon="👤") %}
<ul>
  {% for user in users %}
  <li>
    {% block icon(user_icon, user) %}
      {% if user_icon %}<i class="icon">{{ user_icon }}</i>{% endif %}
    {% endblock %}
    {% block name(user) %}{{ user.name }}{% endblock %}
    {% block default(user) %}<a href="/users/edit/{{user.id}}">Edit</a>{% endblock %}
  </li>
  {% endfor %}
</ul>
{% endmacro %}

{# this uses the default content of all blocks #}
<user-list users="{{users}}"/>
{# rendered #}
<ul><li><i class="icon">👤</i>Aron<a href="/users/edit/1">Edit</a></li></ul>

{# children will be assigned to the default block #}
<user-list users="{{users}}" icon="🧍">
  <span class="disabled">Edit</span>
</user-list>
{# rendered #}
<ul><li><i class="icon">🧍</i>Aron<span class="disabled">Edit</span></li></ul>

{# blocks can be overridden and receive the arguments that are passed #}
<user-list users="{{users}}">
  {% block icon(user_icon, user) %}<i class="icon rounded {{user.color}}">{{user_icon}}</i>{% endblock %}
</user-list>
{# rendered #}
<ul><li><i class="icon rounded red">👤</i>Aron<a href="/users/edit/1">Edit</a></li></ul>

{# when the default block needs arguments it can be called via its name #}
<user-list users="{{users}}">
  {% block default(user) %}<a href="/users/delete/{{user.id}}">Delete</a>{% endblock %}
</user-list>
{# rendered #}
<ul><li><i class="icon">👤</i>Aron<a href="/users/delete/1">Delete</a></li></ul>

{# it would be possible to allow assigning the default block arguments directly but would cause confusion #}
<user-list users="{{users}}" {{ default(user) }}>
  {{ user.name }} {# this works #}
  {% block icon(user_icon) %} {# notice that only the first argument was retrieved #}
    {{ user.name }} {# this doesn't work since user is not defined #}
  {% endblock %}
</user-list>
{# and explicitly calling the default block would still be needed to empty it #}
<user-list users="{{users}}">
  {% block default() %}{% endblock %}
</user-list>
{# rendered #}
<ul><li><i class="icon">👤</i><a href="/users/edit/1">Edit</a></li></ul>

{# shorthand : to avoid {{ }} like in Vue.js and other templating languages might be nice #}
<user-list :users="users">

Using HTML like syntax to call macros might be controversial but I think the result is really neat and leaves the heavy syntax {% %} for control flow more distinct. Syntax wise a similar result could be achieved by leveraging web components but I think the point of server side rendering is to avoid JavaScript which this wouldn't. It's also possible to go all in on web components and make even the macro definitions use HTML syntax via <template> and <slot> but that might be a little too much.

I think this would be a very nice addition to Tera and would allow for seamless use of e.g. Tailwindcss

edit: naming them slot instead of block might still be desirable to avoid ambiguity

[Keats]

That's extremely unlikely to happen. Tera is not only used for rendering HTML so it can't just change its syntax for one usecase. Something like https://jinja.palletsprojects.com/en/3.0.x/templates/#call is likely to be added though.

[ssendev]

Ah of course then that syntax won't fly.

Call is already halfway there but would be improved with the ability to define multiple callers/yield points and to have default content if it's not provided at the call site.

{# the original definition syntax would work but here is an alternate proposal that's closer to call #}
{% macro user_list(users, user_icon="👤") %}
<ul>
  {% for user in users %}
  <li>
    {# a named caller can be defined like this #}
    {{ caller prepend(user) }}
    {# if caller is in {% %} it's a block with default content #}
    {% caller icon(user_icon, user) %}
      {% if user_icon %}<i class="icon">{{ user_icon }}</i>{% endif %}
    {% endcaller %}
    {% caller name(user) %}{{ user.name }}{% endcaller %}
    {% caller(user) %}<a href="/users/edit/{{user.id}}">Edit</a>{% endcaller %}
  </li>
  {% endfor %}
</ul>
{% endmacro %}

{% call(user) user_list(users) %}
  <a href="/users/delete/{{ user.id }}">Delete</a>
{% icon(user_icon, user) %}
  <i class="icon rounded {{ user.color }}">{{ user_icon }}</i>
{% endcall %}
{# rendered #}
<ul><li><i class="icon rounded red">👤</i>Aron<a href="/users/delete/1">Delete</a></li></ul>

To have a library of reusable components macros having these options is extremely useful as can be seen by looking at Vuetify where almost every macro component has multiple callers slots with the most egregious offender probably being v-data-table

[heroin-moose]

A variant of new() that creates empty Tera instance. For example, I'm writing an application that provides templating. The templates are stored in /usr/share/foobar/templates. However, a user can overwrite them by placing templates to /etc/foobar/templates. In order to keep {% include .. %} working the templates are referenced by relative paths. So basically I need two sets of identically named templates which I obtain by using WalkDir and stripping /etc/foobar/templates and /usr/share/foobar/templates.

[Keats]

Isn't that Tera::default()?

[heroin-moose]

Indeed. Thanks, I haven't thought of it.

[rinmyo]

it will be amazing if tera is able to access context. i'm working on KaTeX SSR integration for zola and I really need this feature!

[Keats]

Yep, accessing the context from filters/functions is definitely something I want to add, partly for Zola as well (so we don't do lang=lang etc)

[Piping]

is it possible to add a hex function or hex filter to put a number into hex form?

[nulltier]

A note comment about new hand-written parser isn't clear. @Keats are you going to kick off the pest grammar generated parser?

[jalil-salame]

@jalil-salame https://soupault.app is not really a templating engine, but it's built on using pipes to process data. You might even be able to use it after running Zola, to process code blocks with "lang-dotviz". There's an example here for generating SVGs from Graphviz.

Looks like a good escape hatch, but Zola advertises being a "single binary", if you need a post processing step with a different binary it makes little sense, and you lose the ability to use zola serve to watch for changes and redeploy the website.

[Keats]

There's nothing preventing calling CLI tools in Tera, you just need to define a function that does it.
Zola does not add a function like that because on purpose

[jalil-salame]

There's nothing preventing calling CLI tools in Tera, you just need to define a function that does it.

As far as I can tell, that is part of the Rust API, so you can't call a command from a Template, because that would be a bad idea, right?

Something like

{{ command "date" "+%I" }}

[Keats]

You would need to create your own function eg run_command and pass the args to so a call would look something like `{{ run_command(cmd='date %I') }}. Nothing like that in Tera by default but it would only be a few lines to add to your project if you want to (not in Zola though)

[eguiraud]

throwing it out here in case other people miss this: I would love to see maps become first-class citizens (e.g. having a syntax for map literals and the ability to add/remove keys from an existing map). Related issues: #420 and #673 .

[Keats]

Map literals are supported already in the tera2 repo linked above.
I'm not too sure how to add/remove keys in a simple way from the syntax pov though, vs going through a filter

[eguiraud]

Ah, fantastic, sorry for the noise!

[Keats]

Anyone interested in doing performance improvement: Keats/tera2#4 ? I've tried a few things but it's still slower than tera v1 for the big-table benchmark

[Keats]

https://github.com/Keats/tera2 has made some big progress with perf improvements and nice errors.

The next step is redesigning filters/tests/functions: Keats/tera2#5
Maybe it's going to be like the current ones except we automatically pass a fn to grab data from the current context but curious to see other opinions!

[clarfonthey]

I didn't see it in the issue description or the tera2 readme, but it would be nice to have some way to prevent you from passing extra arguments to macros that aren't used. For example:

{% macro mymacro(one) %}
    {# ... #}
{% endmacro mymacro(one) %}
{{ self::mymacro(one="hello", two="world") }}

Should just be an error. I noticed this when changing the names of arguments and forgot a few calls.

[onkoe]

I'd love to see more static typing in Tera (and, thus, Zola)! Sometimes, it feels a bit unsafe or 'rowdy' to make specific things.

For example, one of my sites has a button macro, somewhat like a component in giant web frameworks. You can place arguments in any order, give incorrect data, or feed it some state that just isn't right.

For this, something like an {{ assert(arg.literal_type == "boolean") }} would be fantastic! Not sure how feasible it is, of course... ☺️

Just my ¢2. Thanks for the hard work on v2 - it's looking nice!

[Keats]

Should just be an error. I noticed this when changing the names of arguments and forgot a few calls.

That's a good idea! Tracking in Keats/tera2#23 but that should be easy to implement.

You can place arguments in any order, give incorrect data, or feed it some state that just isn't right.

The any order is a feature. As for types you do have some tests for it in Tera v1 https://keats.github.io/tera/docs/#built-in-tests but it's not as great.
Is it only for macros? We could potentially add type definition to them but I don't really want to go complex and start having type union, enums, optionals, string literals... It is appealing though

[bemyak]

Not sure how big it is, and even if it should be a part of Tera, but having the ability to add custom filters (and tests/functions?) as WASM plugins would be amazing! It will also (partially at least) solve the long-standing "Zola plugins" topic.

The idea is that you define a simple interface, with string input and output. Libs that implement this interface are compiled into wasm binaries and can be dropped into some folder (/plugins) to be loaded into Tera automatically. Users can share those easily, so we can have a great plugin ecosystem eventually, which covers even the niche use cases without the need to bloat Tera and Zola.

I think Zellij can be a good example of using WASM plugins: https://zellij.dev/documentation/plugin-lifecycle

[Keats]

I was more thinking of using something like lua or rhai than WASM for that usecase. Either way, it's not going to come with v2 straight away, it can be added later.

[clarfonthey]

Since Tera can just add arbitrary Rust code as filters/tests/functions, presumably that could be done as a separate crate/library rather than as part of the engine?

Since it would simply involve creating a wrapper that runs the WASM/Lua code as part of the call function.

[bemyak]

I think that implementing this kind of extension in Rust would be much easier than in Lua/Rhai because of the rich ecosystem. For example, implementing the deflate (de)compression filter took me only a handful LoC and was pretty straightforward. Doing this in Rhai would be much more involved since no ready-made library exists for it.

The other concern is security: do we allow the extensions to read arbitrary files from disk or make network calls? WASM solves this by running modules in a sandbox, and we explicitly grant permissions to them (or so I heard).

[Keats]

The downside of using Rust/wasm is that you get an opaque blob that vs a readable file. Sure it can do less but maybe it's an ok tradeoff?

[RafaelKr]

Twig has a use directive. It's described as Horizontal reuse is a way to achieve the same goal as multiple inheritance, but without the associated complexity:

The documentation describes its use cases and benefits pretty well: https://twig.symfony.com/doc/3.x/tags/use.html

Maybe something that could be taken into consideration for (future) implementation.

[Keats]

Horizontal reuse

That's not going to be added, it seems super niche.

Last call if people have ideas/needs for Keats/tera2#5

[clarfonthey]

One thing that would be nice, but I'm not sure how it would be done, is the ability to make macros that return actual tera values instead of just strings. I know that you can technically JSON encode/decode to get around this, but there isn't actually a JSON decode function in tera by default.

[bemyak]

@clarfonthey, I agree, something like nushell typing would be really nice!

[Keats]

There's no JSON anymore in v2. It would have to be something other than macros though because macros are pretty explicitly only for txt output.
What's the usecase?

[speatzle]

I would like to see some way to handle Enum Variants and their values with if or possibly via a match/switch block.

[clarfonthey]

There's no JSON anymore in v2. It would have to be something other than macros though because macros are pretty explicitly only for txt output. What's the usecase?

In my case, a lot of the macros I made ended up having to deal with multiple inputs/outputs, and this effectively means that things no longer can be macros in this state. Things like:

• Working with date ranges and then displaying them a particular way
• Trying to load values from Zola in a particular way where they were overridden in parent sections, but I couldn't return the result unless it was a string

And in general, you end up with macros being stringly-typed, where you can do things like return numbers, but they all have to be implicitly converted into strings and back at any macro boundary. This also makes refactoring macros difficult because sometimes you just can't split something in half, because you're holding onto more than one, or non-string variables, and you need to have a string to return.

You could argue that all of this is beyond tera's main use cases, but considering how you do have far more than basic functionality (like, for example, handlebars), I'd say it's potentially worth considering. Even a form of macro that simply let's you keep any local variables set when it returns could probably help.

[Keats]

I'm working on the functions (filters etc) and I'm running into some type issues: see PR at Keats/tera2#35

The goal is to define a trait so we can borrow from the context while casting to the right type. Easy example would be a fn escape(&str) filter. I got it to work for owned values but I can't figure out the right incantation of traits to make it work with borrowed ones. If anyone has time to have a look that would be much appreciated. You can revert the last commit to get to where i was.

--

I would like to see some way to handle Enum Variants and their values with if or possibly via a match/switch block.

We don't have the concept of enums in the Value so I'm not sure how that would work

--

Re: macro with Value outputs
I'm not sure about that. I think if you need a lot of logic, it's better to move it out of the template.
Maybe it could be solved with better object manipulation in the language itself?