API: Against a reproducer in a token descriptor

[ruv]

A significant use case for Recognizer API is replacing the system's lexeme translator, a core part of the Forth text interpreter (for example, to produce a different format of code during compilation, e.g. for cross-compilation, see also a discussion in comp.lang.forth in 2020).

Let's consider a Recognizer API that employs token descriptors which are explicitly bases on three components (that are provided when a descriptor is defined).

So, to generate a different format of code during compilation, a program sets the system to use a recognizer that returns descriptors having special these three components.

One of these components is an action that is used by postpone, so a user can be sure that postpone from the host system will work correctly and it will generate code in the new format (according to the descriptor).

But, if the action used by postpone is a reproducer, then the postpone word will fail to generate correct code, since it will use the compile, word from the host system (either directly, or via the auto-generated "postponer").

If the provided action is a postponer (a full postpone action), this scenario will work correctly.

---

The bottom line is that if the API requires a user to provide a reproducer for a new token descriptor (to make postpone applicable), it makes impression that postpone will always work correctly for the corresponding lexeme, but it's wrong in edge cases.

[ruv]

In other words, when the API relies on a reproducer, a postpone action is automated via 1) a reproduce action, 2) a compile action, and 3) the system's compile, word. But the compile action may generate code that is not compatible with what the system's compile, generates. In such cases the correct postpone action cannot be automated by the system.

See also my corresponding comment on forth-standard.