fix: Simplify unicode punctuation

[calculuschild]

Marked version: 5.0.5

Markdown flavor: GitHub Flavored Markdown

Description

Cleans up the unicode punctuation from #2811 by using \p{P} instead of a long list of unicode characters. There are a handful of punctuation characters $+<=>`^|~ not included in that set for whatever reason, so they are still specified here. Includes the accompanying tweaks to a couple other regexes to apply it correctly. This also lets us cover a slightly larger punctuation set since my understanding is JS unicode symbols end at \uFFFF but there are a few more after that.

And a tiny unrelated logic simplification in the emStrong Tokenizer.

My only question is if there is a better way to exclude single characters from \p{P}, for instance in the emStrong, we don't include the current delimiter * or _ in the punctuation checks. I get around this now with an additional lookahead regex:

Something like (?!_)\p{P}

For instance, this example lets us exclude _ from the \p{P} group. I'm ok with this, but if there is some secret "subtraction from a unicode set" syntax, I would like to know.

I didn't add tests, but could potentially look up some of the characters that were missing previously and add them to the existing unicode test.

Contributor

• Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
marked-website	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 7, 2023 5:42pm

[UziTech]

LGTM. I think the current tests are enough to tell if this is working as expected.