Keep buffer for long text during transformation

[qnighy]

In RewritingStream, buffers are sometimes dropped too early to get raw HTML source. It leads to long text node being truncated (only a suffix being emitted).

This PR introduces "buffer safekeeping" extension to keep a buffer at a specified location. It allows RewritingStream to always retrieve correct raw HTML source.

Fixes #292.

[petercmuc]

Hi
I ran into the same problem... what needs to be done in order to have your fix merged?

[fb55]

This is currently blocked by #362. Once that PR is merged, here is the approach I would take instead:

This issue is caused by dropParsedChunk calls in states such as the DATA state. If we delay these calls until after character tokens are emitted, we fix this issue. Because the tokenizer has the indirection of the token queue, we could either (1) move the dropParsedChunk call to the beginning of getNextToken, or (2) remove the token queue by switching to a callback interface.

(1) will always waste a bit of memory, as parsed chunks will only be dropped prior to the next token being generated. We might also want to look into dropping parsed chunks whenever we add to the buffer in the preprocessor, to reduce the impact of this change.

(2) is a much bigger architectural change. It would reliably drop parsed chunks as soon as they are emitted, but might have other unintended consequences (eg. reduced perf).

To fix this specific issue, I would lean towards implementing (1).

[fb55]

Fixed in #432