\markdownInput ignores \input@path and TEXMFPATH

[robertjlee]

Replication

Based on the following extract from 2.2.1 of the manual, I was expected to be able to use \markdownInput on files not in the current directory, as per \input:-

You can use the \markdownInput macro to include markdown documents, similarly
to how you might use the \input TEX primitive to include TEX documents.

So I created 2 files on my system:

/tmp/mbe/mbe.tex

\documentclass{article}
\usepackage{markdown}

\makeatletter
\def\input@path{{/tmp/}}
\makeatother

\begin{document}
\markdownInput{input.md}

\end{document}

/tmp/input.md

# A document

```
Just a document.
Shouldn't these be on separate lines?
```

But when I compile it, it can't open the file via \markdownInput --- although \input (while producing odd-looking output and a message about the unescaped #) works as expected; dumping the contents of input.md as though they were typed in-place.

My command-line is

CWD=/tmp/mbe/ TEXMFPATH=/tmp lualatex \\nonstopmode\\input mbe.tex

Expected behaviour

Both the setting of \input@path, and the setting of TEXMFPATH should be sufficient to locate and open the input file.

Actual behaviour

The following error message is produced, and the process aborts. The same error is generated if the file is completely missing, making me think this is not a permissions issue.

[\direct
lua]:1: could not open file "input.md" for reading
stack traceback:
	[C]: in function 'assert'
	[\directlua]:1: in main chunk.
\markdownLuaExecute ... tex.print(output) end #1 }

Use Case Context

I have a document broken up into smaller files, and I am trying to copy the smaller files to a temporary directory to compile them in isolation to extract metadata from them.

Having to track which documents include which other documents is possible, but expensive, and only required when \markdownInput - rather than \input - is used.

[Witiko]

Hi and thanks for using the Markdown package for TeX.

[... T]he setting of \input@path [...] should be sufficient to locate and open the input file.

I agree that having the \markdownInput command search all directories in the \input@path LaTeX command (or rather the \l_file_search_path_seq expl3 variable) seems both easy to add and also desirable for users.

[... T]he setting of TEXMFPATH should be sufficient to locate and open the input file.

In TeX, opening a file for reading searches the different TEXMF* directory trees automatically. However, the Markdown package for TeX uses Lua to open markdown files, where such behavior is not automatic and requires the explicit use of the KPathsea library. Nevertheless, this change also seems desirable for users.

while producing odd-looking output and a message about the unescaped #

Can you please be more specific? What is odd-looking about the output? What message about unescaped # do you mean?

[robertjlee]

Hi and thanks for using the Markdown package for TeX.

Thank you for making this package; being able to use Markdown in my project will be a significant feature.

[... T]he setting of \input@path [...] should be sufficient to locate and open the input file.

I agree that having the \markdownInput command search all directories in the \input@path LaTeX command (or rather the \l_file_search_path_seq expl3 variable) seems both easy to add and also desirable for users.

I would certainly find it very useful.

while producing odd-looking output and a message about the unescaped #

Can you please be more specific? What is odd-looking about the output? What message about unescaped # do you mean?

All I was trying to say here was that, if you replace \markdownInput with \input, then markdown.md is imported, but parsed by TeX/LaTeX rather than markdown. The first character in the input.md is the # character, which as far as TeX is concerned is an active macro-parameter character not valid in that context, so it discards it with a message to the log and console. The PDF (or DVI) output looks odd because it includes triple open quotes where the backticks are, and the second paragraph all runs onto one line.

I only included this to confirm that the file can be found via \input, i.e. that this is a difference in behaviour between \markdownInput and \input, and not a trivial unrelated problem with the file search path in the test.

[Witiko]

I only included this to confirm that the file can be found via \input, i.e. that this is a difference in behaviour between \markdownInput and \input, and not a trivial unrelated problem with the file search path in the test.

Ok, I was just wondering if you were perhaps suggesting that \input itself should be patched to route markdown files to the Markdown package for TeX. I would not be in favor of such a change because while this would perhaps be convenient to writers, it would change the expected semantics of \input for programmers. Otherwise, the current behavior of \inputting a markdown file should be as expected.

[robertjlee]

Oh, please don't patch \input! Agreed that the current behaviour is as expected; in fact, I even said in the ticket description:

… although \input […] works as expected …

[robertjlee]

I have a partial solution on the TeX side; I am sharing it here in case it's useful to anyone coming across the same problem.

I feel a solution on the Lua side would be simpler and more efficient, with fewer limitations, but unfortunately I don't know Lua.

A new command \inputMd is created, that works as a wrapper for \makdownInput, taking one argument (the filename). It works by searching \input@path, and the current directory, and then executing \markdownInput on each file that is found with a matching filename.

Limitations:

• It will include multiple files if there are multiple files of the given name on the input path (possibly including files like input.md.tex as LaTeX's file macros check for the filename with various extensions added).
• It can't handle cases where one markdown file includes another file, as it will still be looking for the other file in the same directory as the .tex file (I note that some extensions permit this)
• $TEXMFPATH is not supported, only \input@path. (A cursory test shows that the former does not set the latter, at least for lualatex).
• No support for the optional argument to \markdownInput (though this could be trivially added).

\documentclass{article}
\NeedsTeXFormat{LaTeX2e}[2020/02/02]
\usepackage{xinttools}% for xintfor*
\usepackage[utf8]{inputenc}
\usepackage{markdown}

\makeatletter

% Define the TeX search path in \input@path (as used by \input etc):
\def\input@path{{/tmp/}}

% Definition of new command \inputMd{}:

\newread\input@read
\newcommand\inputMd[1]{
  \IfFileExists{#1}{%
    \input@byLines@{#1}
  }{%
    \errmessage{File `#1' missing!}%
  }%
}
\newcommand{\input@processfile}[1]{
  \openin\input@read #1\relax
  \ifeof\input@read
    \closein\input@read
  \else
    \closein\input@read
    \markdownInput{#1}
  \fi
}
\newcommand{\input@byLines@}[1]{
  \xintFor* ##1 in \input@path\do{
    \input@processfile{##1#1}
  }
}
\makeatother

% Example usage
\begin{document}
\inputMd{input.md}
\end{document}

[Witiko]

Thanks for sharing your solution. Here is a somewhat simpler solution using expl3:

\ExplSyntaxOn
\newcommand
  \inputMd
  [ 1 ]
  {
    \file_get_full_name:nNTF
      { #1 }
      \l_tmpa_tl
      {
        \exp_args:NV
          \markdownInput
          \l_tmpa_tl
      }
      {
        \errmessage
          { File~`#1'~missing! }
      }
  }
\ExplSyntaxOff

Function \file_get_full_name:nNTF consults \input@path when searching for the markdown file, see also interface3.pdf.

[Witiko]

@robertjlee In commit 0e80a90 from PR #443, I updated \markdownInput to search for Markdown input files in the directories specified by the expl3 sequence variable \l_file_search_path_seq. In LaTeX, this variable also includes \input@path.

I did not add support for TEXMFPATH, since I was unable to find any documentation for it or reproduce its expected behavior that you described in the original post of this ticket. For example, I created a file example.tex with the following content:

\input input.tex
\bye

Then, I created an empty file test/input.tex and I ran the command TEXMFPATH=test/ luatex example.tex in Bash with TeX Live 2024 installed. However, this produced the following error:

! I can't find file `input.tex'.
l.1 \input input.tex

This leads me to believe that the environmental variable TEXMFPATH is ignored by (Lua)TeX in TeX Live 2024. Can you provide a working example of how TEXMFPATH should function or provide a link to its documentation? If not, I will merge PR #443 as it currently is later this week.

[robertjlee]

Humbly, I was mistaken. Sorry. Having looked at the documentation again, I was confusing $TEXMFPATH with the variable $TEXINPUTS, which may explain why I was having no luck with $TEXMFPATH either.

The issue should have read TEXINPUTS, not TEXMFPATH.

There is no variable $TEXMFPATH that I can find, except for some installers. There is an environment variable $TEXMF, which is a path, but I don't think it's relevant here.

Thank you for fixing it for \input@path; that will be very useful. The documentation I was working from (mostly in case any interested parties want to read further):
https://www.overleaf.com/learn/latex/Articles/An_introduction_to_Kpathsea_and_how_TeX_engines_search_for_files

[Witiko]

Thank you for your response. The example document from #429 (comment) works fine with TEXINPUTS:

$ TEXINPUTS=./test//: luatex example.tex 
This is LuaTeX, Version 1.18.0 (TeX Live 2024) 
 restricted system commands enabled.
(./example.tex (./test/input.tex))
warning  (pdf backend): no pages of output.
Transcript written on example.log.

TEXINPUTS is currently unsupported by \markdownInput. However, that seems OK, since the name TEXINPUTS suggests that the variable should only apply to TeX inputs, not Markdown inputs, especially since Kpathsea also recognizes other environmental variables for different file types such as BIBINPUTS, CLUAINPUTS, CWEBINPUTS, etc.