Consume string token seems to have a bug with newlines

[romainmenke]

csstree/lib/tokenizer/index.js

Line 130 in 4d55558

offset += getNewlineLength(source, offset, code);

That code appears to advance the offset.
It should only return a bad string token and make sure that the next tokenizing pass handles the newline.

https://www.w3.org/TR/css-syntax-3/#consume-string-token

Example :

"fo
o"

[lahmatiy]

Good catch 👍 Agree, a newline must not be a part of bad-string-token. It's a bit tricky to fix it, since serialisation is affected as well. However, I almost done with it, but need to test better.

[romainmenke]

fyi :

the CSS Tokenizer tests I was working on now also have a published corpus on npm : https://www.npmjs.com/package/@rmenke/css-tokenizer-tests

sharing these should make it easier to have multiple high quality css tokenizers :)