In fact, I know it is functionally broken because it fails the existing tests. Still, I'm look for feedback on the approach.

I split what used to be this comment into #201 since it is about master and not this PR.

const RE2 = require('./re2');
const r = new RE2('.', 'g');
const b = Buffer.from('test1test2');
while ((m = r.exec(b))) {
	console.log(m[0].toString(), m.index, r.lastIndex, String.fromCharCode(b[m.index]));
}

t 0 1 t
e 2 2 s
s 4 3 1
t 6 4 e
1 8 5 t
t 10 6
e 12 7
s 14 8
t 16 9
2 18 10

t 0 1 t
e 1 2 e
s 2 3 s
t 3 4 t
1 4 5 1
t 5 6 t
e 6 7 e
s 7 8 s
t 8 9 t
2 9 10 2

I was thinking about a fix like that:

• If it is a buffer: do it like we do now.
• If it is a string: convert it to a buffer. See above.
  • Converting a string to a UTF-8 buffer (and back) is trivial. Node's Buffer supports it directly.

The only problem is to convert offsets from UTF-8 to UTF-16. Plus we need to convert buffers to strings. We can do matchAll() as a post-processing step. So the whole matchAll() can be a JS function. We will need a modified GetUtf16Length() function, but it looks like you already started on that.

Relevant Buffer APIs:

• Static method: Buffer.from(string[, encoding])
• buf.toString([encoding[, start[, end]]])

replaceAll() is a little more involved because of callbacks from C++. That one can be implemented like this:

• If it is a buffer: do it like we do now.
• If it is a string: convert it to a buffer. See above.

The post-processing step should be in the C++ code before/if calling a JS callback. We can indicate it using a custom flag, e.g., using '\b' as an indicator. Checking this boolean flag can be added to the existing code, which does this conversion already (internally re2 works with UTF-8 buffers anyway).

This way the replaceAll() code would include:

const re = new RE2(this, this.flags + '\b');

It will hide the fact that the source is a buffer.

A possible twist: we can do the post-processing for matchAll() in C++ too if it is more efficient. I doubt it'll save much but the proof in the benchmarking. For that, we can use the same '\b' flag.

I hope it makes sense. The idea was to minimize C++ changes yet fix inefficiencies. If readers think that my rambles were incoherent do not hesitate to ask questions and poke holes in the ideas.