Add a build option to duplicate non-page (e.g. images) resources across languages

[bep]

See #11453.

That is mostly a good thing.

But it doesn't work if:

• The multilingual site doesn't have image render hooks.
• The multilingual site have Markup formats (e.g. HTML, Asciidoc) that does not support render hooks.

So, I suggest that we extend the front matter build options to look something like this:

[_build]
  list = 'always'
  publishResources = true
  duplicateResourceFiles = true
  render = 'always'

When duplicateResourceFiles = true we copy non-page resources (e.g. foo.png) for the languages without a dedicated version.

[jmooring]

In support of #11453 and #11540, for multilingual sites with page resources shared between languages, sites will need both link and image render hooks that capture the page resource and use its .RelPermalink.

structure

content/
├── my-bundle/
│   ├── index.de.md
│   ├── index.en.md
│   ├── kitten.en.jpg
│   └── notes.en.pdf
├── _index.de.md
└── _index.en.md

published

├── de/
│   ├── my-bundle/
│   │   └── index.html
│   └── index.html
├── en/
│   ├── my-bundle/
│   │   ├── index.html
│   │   ├── kitten.jpg  <-- de site will need image render hook to get this URL
│   │   └── notes.pdf   <-- de site will need link render hook to get this URL
│   └── index.html
├── favicon.ico
└── index.html

This example contains candidate image and link render hooks, and comprehensive test cases for each.

git clone --single-branch -b hugo-github-issue-11453 https://github.com/jmooring/hugo-testing hugo-github-issue-11453
cd hugo-github-issue-11453
hugo server

layouts/_default/_markup/render-image.html

{{- $u := urls.Parse .Destination -}}
{{- $src := $u.String -}}
{{- if not $u.IsAbs -}}
  {{- with or (.Page.Resources.Get $u.Path) (resources.Get $u.Path) -}}
    {{- $src = .RelPermalink -}}
  {{- end -}}
{{- end -}}
{{- $attributes := dict "alt" .Text "src" $src "title" .Title -}}
<img
  {{- range $k, $v := $attributes -}}
    {{- if $v -}}
      {{- printf " %s=%q" $k $v | safeHTMLAttr -}}
    {{- end -}}
  {{- end -}}
>
{{- /**/ -}}

layouts/_default/_markup/render-link.html

{{- $u := urls.Parse .Destination -}}
{{- $href := $u.String -}}
{{- if not $u.IsAbs -}}
  {{- with or
    ($.Page.GetPage $u.Path)
    ($.Page.GetPage (strings.TrimRight "/" $u.Path))
    ($.Page.Resources.Get $u.Path)
    (resources.Get $u.Path)
  -}}
    {{- $href = .RelPermalink -}}
    {{- with $u.RawQuery -}}
      {{- $href = printf "%s?%s" $href . -}}
    {{- end -}}
    {{- with $u.Fragment -}}
      {{- $href = printf "%s#%s" $href . -}}
    {{- end -}}
  {{- end -}}
{{- end -}}
{{- $attributes := dict "href" $href "title" .Title -}}
<a
  {{- range $k, $v := $attributes -}}
    {{- if $v -}}
      {{- printf " %s=%q" $k $v | safeHTMLAttr -}}
    {{- end -}}
  {{- end -}}
>{{ .Text | safeHTML }}</a>
{{- /**/ -}}

If we add these (or something similar) as built-in templates, we can close #10585.

[bep]

@jmooring yea, I have slept on this, and the hook approach doesn't really work. See my revised proposal above.Does that make sense to you?

[bep]

I have revised the proposal a little:

• Put the setting in front matter
• Make the default behave the way it does today, which means we can remove Don't duplicate page resource output per language #11453, the "noisiest" of the breaking changes, I suspect.

[jmooring]

My thoughts...

Not duplicating page resources is a huge win for multilingual, single host sites. For something this valuable, it would be great if there were a simple on/off switch somewhere.

The challenges with enabling it by default are:

1. The site must use link and image render hooks to capture the markdown destination as a resource.
2. Markdown is the only content format that supports render hooks.

Item 1 doesn't concern me so much; I have advocated for these before in #10585. Proper resolution of markdown link and image destinations has been a challenge for site, theme, and content authors for a very long time. We can handle this initially with documented examples, and perhaps create embedded templates at some point.

With respect to Item 2:

• 99% of sites use markdown only.
• The other 1% may be entirely of one format, or a mix of formats. For example, I've seen sites where one section is markdown, while another is reStructuredText. I've also seen a mix of formats in the same section.

Enabling render hooks for other content formats is a non-starter, so... can we limit resource de-duplication to markdown pages? There's an edge case that looks like this, but probably uncommon:

content/
└── posts/
    └── post-1/
        ├── a.jpg
        ├── index.de.adoc
        ├── index.en.md
        ├── index.es.org
        ├── index.fr.pdc
        ├── index.it.html
        └── index.pt.rst

If we can limit resource de-duplication to markdown pages...

A) Disable the feature by default with an on switch in site config under markup.goldmark (the feature is specific to markdown, and render hooks are limited to Goldmark).

B) Document its usage.

• Are you using link and/or image render hooks? If no, use these examples, if yes, proceed to next step
• Build site and run link checker (e.g., muffet) to verify internal links¹
• If you have related errors, compare your render hooks to the examples...

And we have to do B regardless of how this feature is enabled.

Footnotes

1. A built-in link checker would be useful at some point, as suggested in #5080, perhaps as a CLI flag. ↩

[bep]

can we limit resource de-duplication to markdown pages?

I hadn't thought about that, but that actually makes sense and should be a decent compromise.

content/
└── posts/
    └── post-1/
        ├── a.jpg
        ├── index.de.adoc
        ├── index.en.md
        ├── index.es.org
        ├── index.fr.pdc
        ├── index.it.html
        └── index.pt.rst

That looks ... made up. But if we say:

[markup]
  [markup.goldmark]
        duplicateResourceFiles = false

And somehow, default to true for the others.

It is the owner of the resource that dictates the behaviour; in the example above (assuming the default content language is en), the resource owner is index.en.md. The site owner would need to switch to duplicateResourceFiles = true in Goldmark to make it work for the other formats.

But with that we're back to a situation where we probably need some default render hooks ...

[jmooring]

we probably need some default render hooks

See #11904 (comment) for candidates and test cases. I am very comfortable with these for any new sites, and mostly comfortable for existing sites, but difficult to know for certain (there's some wacky stuff out there).

Multilingual, single host sites with existing render hooks will have to run a link checker before going live to make sure their hooks work as required. See my previous comment.

That looks ... made up

Definitely. I have never seen anything like it.

[bep]

We could probably safeguard this by doing:

1. Only use the default render hooks when len(languages) > 1 and
2. add a way to disable

[markup]
  [markup.goldmark]
        duplicateResourceFiles = false
        disableDefaultRenderHooks = false

[jmooring]

Works for me.

At some point, though, default render hooks for every site would be good. But that's for another day...

[jmooring]

What about something like this instead:

[markup.goldmark.renderhooks.image]
enable = true
multilingualOnly = true

[markup.goldmark.renderhooks.link]
enable = true
multilingualOnly = true

Benefits:

• Granular control (obviously)

• Users can enable as needed/desired for any site

• Sets us up for future capabilities, for example:

  [markup.goldmark.renderhooks.image]
  enable = true
  multilingualOnly = true
  errorLevel = 'warning' # ignore (default), warning, or error (fails the build)
  
  [markup.goldmark.renderhooks.link]
  enable = true
  multilingualOnly = true
  errorLevel = 'warning' # ignore (default), warning, or error (fails the build)
  

[bep]

Yea, that makes sense.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.