Pre-processing text in \markdownInput using lua function

[michaelstepner]

I'm using a custom Lua function to do variable replacement in the style of pandoc-mustache, but performed directly in LuaLaTeX. This provides a convenient way to get variable replacement working in Overleaf, which is important for keeping my coauthors happy (who don't enjoy editing plain text markdown files 😅)

I haven't been able to figure out how to apply my Lua function to pre-process markdown text inserted via \markdownInput{}. Notes:

• it is important that the Lua function is applied before the Markdown processing, because otherwise the plain text characters representing the variable can be modified by the Markdown interpreter. For example, in {{variable}} the left and right braces get rendered.

• I briefly thought I had solved the issue by modifying the delimiters in my luacode and text to [[variable]]. Since square brackets are not rendered by Markdown, [[variable]] will indeed be replaced correctly. But my victory was shortlived, because [[snake_case_variable]] is still rendered and not replaced correctly. So I abandoned this workaround solution, and am hoping to find a way to apply the Lua function to the raw text in the file referenced by \markdownInput{}, before it is rendered by Markdown.

• I wonder if the extensions() option is the right solution. But I haven't been able to try it, due to the issues described in discussion Outdated Markdown package in Overleaf? #251. I'm also not sure what the order of operations would be on processing extensions()

Minimal example

variable-replacement.tex

\documentclass{article}

\usepackage{luacode}
\begin{luacode}
function mustache_variable_replacement ( s )
  s = s:gsub ( "{{%s*variable%s*}}" , "successful test" )
  return s
end
\end{luacode}
    
\AtBeginDocument{%
  \luaexec{luatexbase.add_to_callback("process_input_buffer", mustache_variable_replacement, "mustache_variable_replacement")}%
}

\usepackage{markdown}
\begin{document}

\begin{markdown}
This Markdown text written in the `{markdown}` LaTeX environment contains a {{variable}}.
\end{markdown}

\markdownInput{test.md}

\end{document}

test.md

This Markdown text written inserted using `\markdownInput` contains a {{variable}}.

Typeset Output:

[Witiko]

It seems to me that you are missing a % before the s* in the following line:

s = s:gsub ( "{{s*variable%s*}}" , "successful test" )

Otherwise, your solution should work. If you'd like your solution to work with the \markdownInput command as well and can grok using LPEG instead of regular expressions, then writing a user-defined syntax extension may be more appropriate.

Another solution would be to redefine the macro that renders left braces and do the variable expansion in TeX. This will also work with both \begin{markdown} ... \end{markdown} and \markdownInput, similarly to syntax extensions.

[michaelstepner]

It seems to me that you are missing a % before the s*

You're right! That didn't matter in my example because I didn't have any spaces before the variable name, but I've edited my original post to fix the bug.

your solution should work. If you'd like your solution to work with the \markdownInput command as well and can grok using LPEG instead of regular expressions, then writing a user-defined syntax extension may be more appropriate.

@Witiko I haven't worked with LPEG before, could you point me to some information or an example on using LPEG to construct a user-defined extension?

Another solution would be to redefine the macro that renders left braces and do the variable expansion in TeX. This will also work with both \begin{markdown} ... \end{markdown} and \markdownInput, similarly to syntax extensions.

I did test that solution, but rejected it because I'd need to modify how italics are rendered as well (pasting this bullet from the original post):

• I briefly thought I had solved the issue by modifying the delimiters in my luacode and text to [[variable]]. Since square brackets are not rendered by Markdown, [[variable]] will indeed be replaced correctly. But my victory was shortlived, because [[snake_case_variable]] is still rendered and not replaced correctly. So I abandoned this workaround solution, and am hoping to find a way to apply the Lua function to the raw text in the file referenced by \markdownInput{}, before it is rendered by Markdown.

@Witiko do you think there is an easy way to modify how braces and italics are rendered that would generically make {{ variables_like_this }} render correctly?

[Witiko]

I haven't worked with LPEG before, could you point me to some information or an example on using LPEG to construct a user-defined extension?

For an introduction to LPEG, see the article "Parsing complex data formats in LuaTeX with LPEG" by Henri Menke. For a reference manual, see the LPEG docs. To see how Markdown extensions work, see Section 2.1.2 (User-Defined Syntax Extensions) in the technical documentation of the Markdown package. I can write the syntax extension for you.

do you think there is an easy way to modify how braces and italics are rendered that would generically make {{ variables_like_this }} render correctly?

You would need to use the underscores = false option before it would be feasible. Even then, it would be quite brittle. It would be better to either use syntax extensions for this, or use TeX macros in conjunction with the hybrid option:

\documentclass{article}
\usepackage{markdown}
\def\variable{world}
\begin{markdown*}{hybrid = true}
Hello, \variable!
\end{markdown*}

[Witiko]

@michaelstepner I wrote an extension for you:

variables.lua

local variables = {
  api_version = 2,
  grammar_version = 2,
  finalize_grammar = function(reader)
    local any = lpeg.P(1)
    local dollar = lpeg.P("$")
    local left_brace = lpeg.P("{")
    local right_brace = lpeg.P("}")

    local read_variable = dollar 
                        * left_brace
                        * lpeg.C((any - right_brace)^1)
                        * right_brace
                        / function(name)
                          local value = tostring(markdown_variables[name])
                          return value
                        end

    reader.insert_pattern("Inline before Symbol", read_variable, "Variables")
    reader.add_special_character("$")
  end
}

return variables

document.tex

\documentclass{article}
\usepackage{soul}
\directlua{
   markdown_variables = {
      value_of_pi = math.pi,
      variable = "vehicle",
      ["1 + 2"] = 3,
    }
}
\usepackage[extension = strike-through.lua,
            extension = variables.lua,
            hybrid = true]{markdown}
\begin{document}
\begin{markdown}
This is //a lunar roving ${variable}// strike-through text.

$$ 1 + 2 = ${1 + 2} $$
$$ \pi = ${value_of_pi} $$
\end{markdown}
\end{document}

You can try it out on Overleaf.