It would be nice to be able to specify a custom tokenizer function in the fork API, so that CSSTree could be extended to syntaxes that don't follow the spec at all points. For example, Adblock Extended CSS syntax, where some psudeo classes don't follow the CSS spec, and bracket balancing or other things need to be checked, similar to how the CSS spec declares bad-url-token. Two typical problems:

• :contains(a'b) tokenized as unclosed string
• :xpath(//*[contains(text(),"a")]) tokenized as unclosed comment

These problems can be solved by the customizability of the parser via the fork API, but it would be a more efficient solution if the tokenizer could be directly customizable at some level.

This way, the default tokenizer in CSSTree can still strictly follow the CSS spec, but the forks could be more flexible.

I thought of a very simple thing:

import * as cssTree from 'css-tree';

// this function should be compatible with the default tokenizer
// and should follow the type of basic tokens:
const customTokenize = function(source, onToken) {
    // ...
}

const fork = cssTree.fork({
    // Use the customized tokenizer
    tokenize: customTokenize,
});