Core: Fixed greedy rematching reach bug

[RunDevelopment]

The bug fixed by this PR caused Prism to stop rematching too early which caused incorrect results (see #2694).

The bug was caused by rematching reach. Rematching reach was introduced in #2032 and one of its main features is that the area (=reach) in which Prism will redo the tokenization can be extended. Extending the reach is necessary for correctness but may degrade performance a little.

The cause of this bug is that nested rematching did not extend the caller's reach. Rematching is done recursively and if a recursive call extends the reach, the caller's reach has to be extended as well. The fix is the do exactly that.

Performance impact:

None, surprisingly. I expected that the fixed version would be slower since we might do more rematching but that wasn't the case. I don't know why thou.

Benchmark:

(local refers to the fixed version in this branch.)

Found 9 cases with 27 files in total.
Test 2 candidates with Prism.tokenize
Estimated duration: 9m 0s

------------------------------------------------------------

c

  https://raw.githubusercontent.com/git/git/master/mergesort.c (1 kB)
  | local              0.08ms ±  0%  182smp 1.01x
  | PrismJS@master     0.08ms ±  0%  177smp
  https://raw.githubusercontent.com/git/git/master/mergesort.h (1 kB)
  | local              0.02ms ±  0%  180smp
  | PrismJS@master     0.02ms ±  0%  183smp 1.01x
  https://raw.githubusercontent.com/git/git/master/remote.c (66 kB)
  | local              2.98ms ±  0%  170smp
  | PrismJS@master     3.00ms ±  0%  161smp 1.01x
  https://raw.githubusercontent.com/git/git/master/remote.h (11 kB)
  | local              0.37ms ±  0%  182smp
  | PrismJS@master     0.37ms ±  0%  182smp 1.00x

------------------------------------------------------------

css

  ../assets/style.css (7 kB)
  | local              0.88ms ±  0%  183smp
  | PrismJS@master     0.89ms ±  0%  180smp 1.01x

------------------------------------------------------------

css!+css-extras (css)

  ../assets/style.css (7 kB)
  | local              1.46ms ±  0%  181smp
  | PrismJS@master     1.47ms ±  0%  180smp 1.01x

------------------------------------------------------------

javascript

  ../assets/utopia.js (11 kB)
  | local              1.48ms ±  0%  178smp
  | PrismJS@master     1.49ms ±  0%  177smp 1.01x
  ../components.json (28 kB)
  | local              4.01ms ±  0%  172smp 1.02x
  | PrismJS@master     3.94ms ±  0%  174smp
  ../package-lock.json (266 kB)
  | local             26.02ms ±  0%  119smp
  | PrismJS@master    27.93ms ±  1%  117smp 1.07x
  https://cdnjs.cloudflare.com/ajax/libs/prism/1.20.0/prism.js (29 kB)
  | local              3.84ms ±  0%  165smp 1.00x
  | PrismJS@master     3.83ms ±  0%  166smp
  https://cdnjs.cloudflare.com/ajax/libs/prism/1.20.0/prism.min.js (14 kB)
  | local              3.61ms ±  0%  165smp
  | PrismJS@master     3.83ms ±  0%  166smp 1.06x
  https://code.jquery.com/jquery-3.4.1.js (274 kB)
  | local             38.90ms ±  0%   84smp
  | PrismJS@master    42.60ms ±  6%   78smp 1.10x
  https://code.jquery.com/jquery-3.4.1.min.js (86 kB)
  | local             23.59ms ±  1%  102smp 1.02x
  | PrismJS@master    23.08ms ±  1%  105smp

------------------------------------------------------------

json

  ../components.json (28 kB)
  | local              3.17ms ±  1%  169smp
  | PrismJS@master     3.23ms ±  1%  164smp 1.02x
  ../package-lock.json (266 kB)
  | local             19.61ms ±  1%  124smp
  | PrismJS@master    20.67ms ±  1%   97smp 1.05x

------------------------------------------------------------

markup

  ../download.html (4 kB)
  | local              0.44ms ±  1%  171smp
  | PrismJS@master     0.44ms ±  0%  176smp 1.01x
  ../index.html (20 kB)
  | local              2.36ms ±  1%  169smp
  | PrismJS@master     2.38ms ±  1%  167smp 1.01x
  https://github.com/PrismJS/prism (194 kB)
  | local             21.62ms ±  1%  113smp
  | PrismJS@master    21.93ms ±  1%  111smp 1.01x

------------------------------------------------------------

markup!+css+javascript (markup)

  ../download.html (4 kB)
  | local              0.69ms ±  1%  172smp 1.01x
  | PrismJS@master     0.68ms ±  1%  174smp
  ../index.html (20 kB)
  | local              2.57ms ±  0%  177smp 1.01x
  | PrismJS@master     2.55ms ±  1%  172smp
  https://github.com/PrismJS/prism (194 kB)
  | local             21.37ms ±  1%  113smp 1.02x
  | PrismJS@master    20.98ms ±  0%  116smp

------------------------------------------------------------

ruby

  https://raw.githubusercontent.com/rails/rails/master/actionview/lib/action_view/base.rb (11 kB)
  | local              0.63ms ±  0%  181smp 1.01x
  | PrismJS@master     0.63ms ±  0%  180smp
  https://raw.githubusercontent.com/rails/rails/master/actionview/lib/action_view/layouts.rb (16 kB)
  | local              0.86ms ±  0%  182smp 1.01x
  | PrismJS@master     0.86ms ±  0%  183smp
  https://raw.githubusercontent.com/rails/rails/master/actionview/lib/action_view/template.rb (13 kB)
  | local              0.83ms ±  0%  182smp 1.01x
  | PrismJS@master     0.82ms ±  0%  182smp

------------------------------------------------------------

rust

  https://raw.githubusercontent.com/rust-lang/regex/master/src/compile.rs (42 kB)
  | local              5.27ms ±  0%  166smp 1.01x
  | PrismJS@master     5.24ms ±  0%  167smp
  https://raw.githubusercontent.com/rust-lang/regex/master/src/lib.rs (28 kB)
  | local              0.47ms ±  0%  179smp
  | PrismJS@master     0.48ms ±  0%  183smp 1.01x
  https://raw.githubusercontent.com/rust-lang/regex/master/src/utf8.rs (9 kB)
  | local              1.12ms ±  0%  181smp
  | PrismJS@master     1.12ms ±  0%  180smp 1.00x

------------------------------------------------------------

summary
                  best  worst  relative
  local             16     11     1.00x
  PrismJS@master    11     16     1.01x

---

This fixes #2694.

[github-actions]

JS File Size Changes (gzipped)

A total of 1 files have changed, with a combined diff of +16 B (+0.5%).

file	master	pull	size diff	% diff
components/prism-core.min.js	3.01 KB	3.03 KB	+16 B	+0.5%

Generated by 🚫 dangerJS against eb1e350