How to convert tokens to unescaped, usable values?

[romainmenke]

css-tree/tokenizer exposes tokens as [type, start, end].
But it does not seem to expose unescaped and usable values.

For example :

https://www.w3.org/TR/css-syntax-3/#consume-numeric-token

A dimension-token should have a value, a unit and a type flag.

10px:

• value: 10
• unit: px
• type: integer

10\70 x:

• value: 10
• unit: px
• type: integer

Are there helper functions for this?

[romainmenke]

Another example:

Does a hash token have type id or unrestricted.
The only way to answer this is by checking if it starts with ident start code points or ident code points.

[lahmatiy]

CSSTree tokenizer doesn't produce an object for a token for performance/memory reasons. That's not necessary in most cases. However, it provides some helper functions to decode/encode values for ident, string and url tokens. See Util functions for detail.

[romainmenke]

Thank you for that info!
I am using those helper functions but they are limited.

Is there any way to collect the information outlined above with CSSTree?
Is there a different API that does produce an object per token?

This information is needed to correctly parse CSS so I assume it exists somewhere in CSSTree?

[chinchang]

Another method is passing the output from css-tree to something like lightningcss. Works well!

[romainmenke]

Another method is passing the output from css-tree to something like lightningcss. Works well!

I am sure it does, I can also pass it to one of the parsing utilities I've created in @csstools

But for users this doesn't make sense.
Why would they bloat their node_modules and pay the performance penalty of tokenizing/parsing twice?

---

csslex strikes the right balance.

It tokenizes in the same minimal way as csstree but it also exposes a value function to gather the information I listed above.

Having such a function in csstree would be useful.