Deal with invisible chars

[markelog]

Like BOM - https://en.wikipedia.org/wiki/Byte_order_mark, now they marked as a whitespace, it seems we just need to miss them

[mattiasrunge]

+1 to get this fixed, it used to work fine before upgrading to version 3

[cashburn]

I would also like to see this fixed, although I believe it's partially a problem in Babylon. In Babylon, they tokenize the JS, but if there is a BOM at the beginning of the file, the first token starts at index 1. Then in cst (elementTree.js: 163), anything between the last token (nothing) and the next token (first token) is made into a whitespace token. So there's a whitespace token with only an invisible character as the first token, then the next token is where it actually starts. So, at that point, cst could ignore/throw away the BOM, label it as another token, or keep labeling it as whitespace and let the users deal with it. I almost made a pull request, but I wasn't sure what the devs wanted to do about it, so I ended up just making a temporary change in JSCS, referenced in the JSCS issue above.

[markelog]

Sounds like you figure it out - send a pull request :) Regards, Oleg

…

On 20 Jul 2017, at 16:28, Colin Ashburn ***@***.***> wrote: I would also like to see this fixed, although I believe it's partially a problem in Babylon. In Babylon, they tokenize the JS, but if there is a BOM at the beginning of the file, the first token starts at index 1. Then in cst (elementTree.js: 163), anything between the last token (nothing) and the next token (first token) is made into a whitespace token. So there's a whitespace token with only an invisible character as the first token, then the next token is where it actually starts. So, at that point, cst could ignore/throw away the BOM, label it as another token, or keep labeling it as whitespace and let the users deal with it. I almost made a pull request, but I wasn't sure what the devs wanted to do about it, so I ended up just making a temporary change in JSCS. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.