Make i18n extractor execution idempotent

[acalvo]

🚀 feature request

Relevant Package

This feature request is for @angular/localize

Description

Each time you run ng xi18n --ivy, the output changes, as the order of the trans-units is different from execution to execution.

Describe the solution you'd like

The output could be sorted (in any way) to ensure that if the source code doesn't change, the output is always the same. This would make it easier to see the changes in the version control system, etc.

Describe alternatives you've considered

Meanwhile, we are ordering the resulting XLIFF by ourselves.

[petebacondarwin]

Can I assume that if you do not change the source code then the extracted output is indeed the same from one run to the next? And that the changes to the ordering you see are due to changes to the source code?

[acalvo]

Can I assume that if you do not change the source code then the extracted output is indeed the same from one run to the next? And that the changes to the ordering you see are due to changes to the source code?

No, it isn't, and that's our main problem...

With no changes to the source code I run ng xi18n --ivy and get a XLIFF output. Ok. I run it again, and get the a different output (same messages, same IDs, but different order).

[petebacondarwin]

Hmm, I guess this must be due to the async webpack build in the CLI, that can cause source files to be compiled in different orders...

[petebacondarwin]

I'm going to see if I can sort the messages by the file that they are in....

[suer85]

@acalvo how are you ordering the result yourself?

[acalvo]

@suer85 once Angular generates the XLIFF, we run a script to sort it an replace the file generated by Angular.

This sort-xlf.ts works for us:

import * as fs from 'fs';
import * as libxmljs from 'libxmljs';
import * as path from 'path';

const i18nDir = process.argv[2];
const ns = 'urn:oasis:names:tc:xliff:document:1.2';

const xmlDoc = libxmljs.parseXml(fs.readFileSync(path.join(i18nDir, 'translation.en.xlf')).toString());
const body = xmlDoc.get('//ns:body', { ns });
const transUnits = body.childNodes() as Array<libxmljs.Element>;
const newBody = new libxmljs.Element(xmlDoc, 'body', '');
newBody.addChild(transUnits[0]);
transUnits.filter(x => x.type() === 'element').sort((a, b) => {
  return parseInt(a.attr('id').value(), 10) - parseInt(b.attr('id').value(), 10);
}).forEach(t => {
  newBody.addChild(t);
});
newBody.addChild(transUnits[transUnits.length - 1]);
body.replace(newBody);
fs.writeFileSync(path.join(i18nDir, 'translation.en.xlf'), xmlDoc.toString(), 'utf8');

And execute it this way: ng extract-i18n --ivy --output-path $I18N_DIR --out-file translation.en.xlf --progress false && npx ts-node sort-xlf.ts $I18N_DIR"

Hope it helps as a workaround till it's properly solved by the Angular team :)

[petebacondarwin]

Working on it right now!

[suer85]

@suer85 once Angular generates the XLIFF, we run a script to sort it an replace the file generated by Angular.

This sort-xlf.ts works for us:

import * as fs from 'fs';
import * as libxmljs from 'libxmljs';
import * as path from 'path';

const i18nDir = process.argv[2];
const ns = 'urn:oasis:names:tc:xliff:document:1.2';

const xmlDoc = libxmljs.parseXml(fs.readFileSync(path.join(i18nDir, 'translation.en.xlf')).toString());
const body = xmlDoc.get('//ns:body', { ns });
const transUnits = body.childNodes() as Array<libxmljs.Element>;
const newBody = new libxmljs.Element(xmlDoc, 'body', '');
newBody.addChild(transUnits[0]);
transUnits.filter(x => x.type() === 'element').sort((a, b) => {
  return parseInt(a.attr('id').value(), 10) - parseInt(b.attr('id').value(), 10);
}).forEach(t => {
  newBody.addChild(t);
});
newBody.addChild(transUnits[transUnits.length - 1]);
body.replace(newBody);
fs.writeFileSync(path.join(i18nDir, 'translation.en.xlf'), xmlDoc.toString(), 'utf8');

And execute it this way: ng extract-i18n --ivy --output-path $I18N_DIR --out-file translation.en.xlf --progress false && npx ts-node sort-xlf.ts $I18N_DIR"

Hope it helps as a workaround till it's properly solved by the Angular team :)

Thanks :)

[JonasSamuelsson]

Do you have an ETA for a release where this fix is included?

[petebacondarwin]

@JonasSamuelsson - the PR is marked for merging into "patch" so it will appear in 11.0.6 very early in 2021.

[minijus]

Is it already very early in 2021? :)

[petebacondarwin]

Very very early! Happy New Year.

[acalvo]

Thanks for the fix @petebacondarwin ! 😄

Just a quick clarification:

the PR is marked for merging into "patch" so it will appear in 11.0.6 very early in 2021.

This didn't happen, did it? I'm seeing in the Angular changelog that it landed in 11.1.0-next.4 but not in 11.0.6. And the changelog seems to be OK because I just tried it and the old behavior keeps happening in 11.0.6.

This is not a complain - I just wanted to prevent confusion to the people following this issue, as it seems that it'll be included in the 11.1.x branch and not in the 11.0.x line :).

[petebacondarwin]

Hi @acalvo - it turned out that there was a conflict in merging that PR into the patch branch. Since we are about to release 11.1.0 RC we decided that it was simplest to only merge the PR into the master branch, which means that the fix will be available in 11.1.0. Sorry about the confusion.

[acalvo]

Perfect! No worries - I'm sure we all can wait a few weeks till 11.1.0 goes stable 😊

Thanks again.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}