Whitespace processing

[thekip]

The whitespace processing in messages is the big balck box in lingui. When i originally started working on the project the normalizeWhitespaces function was even not covered by tests directly. So there no a specific spec how this should work actually.

In general what we have now it seems works, but there are sometimes cases which processed incorrectly by this normalizing.

I originally thought that processing is not needed at all and the safest way would be just leave string as it is. Yes, the message in catalog would not be so tidy as now, but at least this will resolve all kind of problems and mismatched related to whitespaces.

But that will not work with experimental-extractor (#1797) the esbuild re-print the AST, so original formatting is not preserved and it means runtime message would mismatch to what extracted to catalog.

By the way, esbuild when re-printing doing it safe (or lossless), so the final html result on the page would not be changed. So why we could not implement similar rules for whitespace normalizing to make it follow some spec?

I've found detailed whitespace explanation here https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Whitespace#how_does_css_process_whitespace

And I think we can implement exactly the same logic as described for non-block context.

The current whitespace normalization is a bit diffrent:

// replace whitespace before/after newline with single space
const keepSpaceRe = /\s*(?:\r\n|\r|\n)+\s*/g
// remove whitespace before/after tag or expression
const stripAroundTagsRe =
  /(?:([>}])(?:\r\n|\r|\n)+\s*|(?:\r\n|\r|\n)+\s*(?=[<{]))/g

export function normalizeWhitespace(text: string): string {
  return (
    text
      .replace(stripAroundTagsRe, "$1")
      .replace(keepSpaceRe, " ")
      // keep escaped newlines
      .replace(/\\n/g, "\n")
      .replace(/\\s/g, " ")
      // we remove trailing whitespace inside Plural
      .replace(/(\s+})/gm, "}")
      // we remove leading whitespace inside Plural
      .replace(/({\s+)/gm, "{")
      .trim()
  )
}

And here are examples:

You have

          {count, plural, one {Message} other {Messages}}
↓ ↓ ↓ ↓ ↓ ↓
You have{count, plural, one {Message} other {Messages}}

in example above it has a broken message have{count,

    Hello <strong>World!</strong><br />
    <p>
     My name is <a href="/about">{{" "}}
      <em>{{name}}</em></a>
    </p>
↓ ↓ ↓ ↓ ↓ ↓
Hello <strong>World!</strong><br /><p>My name is <a href="/about">{{" "}}<em>{{name}}</em></a></p>

{count, plural, one {

              One hello

            } other {

              Other hello

            }}
↓ ↓ ↓ ↓ ↓ ↓
{count, plural, one {One hello} other {Other hello}}

It is also broken in this example:

"\n" +
          "        Wonderful framework \n" +
          "        <0>Next.js</0>\n" +
          "         say hi. And \n" +
          "        <1>Next.js</1>\n" +
          "         say hi.\n" +
          "      "

↓ ↓ ↓ ↓ ↓ ↓

"Wonderful framework <0>Next.js</0>say hi. And <1>Next.js</1>say hi."

What do you think? Do you have other ideas how to make this whitespace trimming mechanism more safe and stable?

Related Issues:
#1799
#1797
#1834

[thekip]

@andrii-bodnar @vonovak fyi

[andrii-bodnar]

I just read the suggested article "How whitespace is handled by HTML, CSS, and in the DOM", and their approach looks pretty clear and straightforward to me. I think it's a good idea to follow it.

I'm concerned about how to define which tags are inline and which are block. There could be a lot of custom tags from different frameworks or custom web components, and it could be difficult to determine which are inline and which are block.

[thekip]

I'm concerned about how to define which tags are inline and which are block. There could be a lot of custom tags from different frameworks or custom web components, and it could be difficult to determine which are inline and which are block.

Yes, that's why i propose to treat all strings as they in the "non block" context.

A actually beleive that this normalization rules could be even more simple:

1. Replace all tabs with spaces
2. Replace all line breaks with spaces
3. Replace all multiple spaces with one space
4. Trim the result

Take this example:

 Hello <strong>World!</strong><br />
    <p>
     My name is <a href="/about">{{" "}}
      <em>{{name}}</em></a>
    </p>
// ↓ ↓ ↓ ↓ ↓ ↓
// Current logic
Hello <strong>World!</strong><br /><p>My name is <a href="/about">{{" "}}<em>{{name}}</em></a></p>

// Proposed Logic
Hello <strong>World!</strong><br />◦<p>◦My name is <a href="/about">{{" "}}◦<em>{{name}}</em></a>◦</p>

[andrii-bodnar]

The current logic in this example looks more translator-friendly: there are no extra spaces after the <br />, <p>, and </a> tags that should be translated (these space characters also will be considered in reporting, for example).

Also, why do we need the {{" "}} here if we already have a space in the My name is part?

[thekip]

...friendly and broken, as we see in this example

"\n" +
          "        Wonderful framework \n" +
          "        <0>Next.js</0>\n" +
          "         say hi. And \n" +
          "        <1>Next.js</1>\n" +
          "         say hi.\n" +
          "      "

↓ ↓ ↓ ↓ ↓ ↓

"Wonderful framework <0>Next.js</0>say hi. And <1>Next.js</1>say hi."

Also, why do we need the {{" "}} here if we already have a space in the My name is part?

i have no idea, i just packed up a real test case from the suite

[thekip]

#1882

[thekip]

I'm also considering completely disabling whitespace processing in js macros, since it's not needed there at all.