[JSON] Rewrite syntax #3097

jrappen · 2021-10-21T00:01:24Z

Using inheritance split up JSON.sublime-syntax into:
- JSON.sublime-syntax with scope:source.json
- JSONC.sublime-syntax with scope:source.json.jsonc
- JSON5.sublime-syntax with scope:source.json.json5
- JSON_dotNET.sublime-syntax with scope:source.json.json-dotnet
Add many more file extensions for JSON & JSONC:
- add doc links to extensions where applicable as a reference to be
  able to more quickly verify that they (still) use said syntax
  flavor
JSON:
- Make use of newer syntax features including those only available
  in version: 2 syntaxes
- Make use of variables (with optimizations provided by @deathaxe
  and regex patterns provided by @Thom1729)
- Context names now more closely match the naming scheme of other
  (recently re-written) default syntaxes
- (correctly formatted) JSON code can now be prettified or minified
  via the context menu or the command palette. JSON code can
  optionally be auto-prettified on pre save events.
- highlight leading, trailing & multiple commas as invalid
- only allow exactly one structure (object, array) or value
  (constant, number, string) at top level (thanks to @keith-hall)
- links (meta.link.inet) and email addresses (meta.link.email) are
  scoped the same as in Markdown (thanks to @deathaxe)
JSONC:
- highlight some files by default as JSONC (as decided by
  @jskinner in Separate JSON and "JSON with comments" syntaxes? #285)
- highlight leading & multiple commas as invalid, trailing as valid
- scope empty block comments as such
- support syntax based folding of ST4131+,
  compare [JSON] Add support for syntax based folding #3291
JSON5:
- explicitly pos numbers, hexadecimal ints, Infinity and NaN
- single quoted strings
- more escape chars for strings
- ECMA identifierName as object keys (regexes thanks to @Thom1729)
  - scoped as plain unquoted strings (thanks to @Thom1729)
  - support string interpolation (thanks to @deathaxe)
- line continuation in strings (with tests thanks to @keith-hall)
JSON.NET:
- support requested by @keith-hall,
  built with feedback from @michaelblyons
Objects:
- Highlighting speed improvements for empty objects (thanks to
  @FichteFoll)
- Make mapping.* contexts more modular
Arrays:
- Highlighting speed improvements for empty arrays (thanks to
  @FichteFoll)
Numbers:
- Correctly scope number signs with constant.numeric.sign instead
  of keyword.operator.arithmetic
- Significantly improve number highlighting (thanks to @deathaxe)
Completions:
- completions have been added for language constants, including kind info
  and details (with links to docs)
  - null, false, true for JSON
  - Infinity and NaN for JSON5
Settings:
- a default_extension is now set for all JSON flavors
Symbol index:
- with an object structure at the top-level, only top-level keys
  within now show up in the index (including tests for symbols and
  syntax)
Tests:
- test files now test the base scope
- Significantly extend tests to cover more parts of the syntaxes
- Split original test file into logical parts
- Add indentation tests for:
  - json, json5 & jsonc
  - mapping (objects), sequence (arrays)
- Add symbols tests for:
  - top-level keys of object structures (thanks to deathaxe)
  - languages: json, json5 & jsonc
- Fix tests for meta.mapping meta.mapping.*
Leave JSON headers in Markdown as json only, but split up fenced
code blocks into json, json5 & jsonc to behave similarly to
GitHub Flavored Markdown

BREAKING CHANGES:

JSON does not have values that can be set via an inline calculation
with the help of operators, but only simple number values. Scopes for
number signs have changed from being keyword.operator.arithmetic to
constant.numeric.sign. Color scheme authors should add this, should
it be missing.
The JSON.sublime-syntax now marks comments as invalid, third party
plugin authors should instead target JSONC.sublime-syntax to keep
the user experience as-is.
Indexed symbols (i.e. top-level keys in JSON object structures) are
scoped as source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key).
Color scheme authors should add special highlighting to differentiate
them from other keys.
fix Separate JSON and "JSON with comments" syntaxes? #285
address Scope for keys in mappings/dictionaries (e.g. JSON, YAML) #421 (thanks to @FichteFoll)
address Remove Incompatible Regex Patterns #481 to remove incompatible regex patterns according
to @wbond
address Syntax Development Tips/Advice #757 to fix line comments for JSONC (thanks to
@keith-hall)
address [RFC] Sublime Syntax Main Key Sort Order #2430 using sort-order (as requested by @deathaxe)
address [RFC] "Empty" values #2711 with regards to constant.language.null
vs. constant.language.empty (thanks to @FichteFoll)
address [RFC] Use punctuation.definition for brackets defining collection literals rather than punctuation.section #2852 to fix scopes of curly braces & square
brackets in JSON (thanks to @Thom1729)
address [RFC] Key-Value Separators #3228 to fix punctuation.separator scopes,
compare [JSON] Rename mapping separator scopes #3270
address [Symbol Index] Similar test tool to st_syntax_tests for testing Symbol Lists sublime_text#3154 and add symbol tests

jrappen · 2021-10-21T00:02:06Z

⚠️ This is a first step to show my current work in progress.

There is still a lot of work to do. Please comment below with suggestions.

jrappen · 2021-10-21T00:04:24Z

As @jskinner requested in #285, this PR applies JSONC to *.json files by default to not confuse beginners too much who do not know how to set a syntax for a view.

michaelblyons · 2021-10-21T00:27:25Z

What do you think about using source.json and source.json.jsonc? I worry a little that moving the primary *.json association to source.jsonc will break a bunch of people and packages' configuration scopes.

jrappen · 2021-10-21T00:37:11Z

Sounds reasonable.

I haven't looked at other packages that either use JSON or are affected by this change, yet.

jfcherng · 2021-10-21T00:39:19Z

I haven't looked at other packages that either use JSON or are affected by this change, yet.

Like I have one in my custom keybindings via selector context.

    // plugin: JsPrettier
    {
        "keys": ["ctrl+alt+f"],
        "command": "js_prettier",
        "context": [
            {
                "key": "selector",
                "operator": "equal",
                "operand": "text.html.basic | text.html.markdown | text.html.vue | source.js | source.json | source.css | source.scss | source.less | source.ts | source.tsx | source.yaml"
            }
        ]
    },

jrappen · 2021-10-21T00:52:55Z

Ok, follow up question, should I change stuff like:

scope: comment.line.double-slash.jsonc

to

scope: comment.line.double-slash.json.jsonc

for JSONC.sublime-syntax?

michaelblyons · 2021-10-21T01:07:14Z

Ok, follow up question, should I change stuff like:
scope: comment.line.double-slash.jsonc
to
scope: comment.line.double-slash.json.jsonc
for JSONC.sublime-syntax?

That's disputed. One camp says "yes." One camp says "no." One camp actually says "Change the last . to a -."

In this instance, I am in the "keep it as .jsonc" camp. Closest thing in default packages is the GitHub markdown extensions.

deathaxe · 2021-10-21T16:39:01Z

Agree. The backward compatibility concerns are mainly tarteting the syntax's main scope here.

If a derived/extending syntax adds more patterns, it's ok to only add it's trailing part of a main scope.

So I am also in the "keep it as .jsonc" camp.

deathaxe · 2021-10-21T16:49:19Z

JSON with Comments (jsonc) is just an extension of JSON. I don't therefore see any value in adding a JSON (Basic) which JSON is derived from by just adding some file extensions.

JSON/JSONC.sublime-syntax

JSON/JSON (Basic).sublime-syntax

jrappen · 2021-10-21T16:52:56Z

JSON with Comments (jsonc) is just an extension of JSON. I don't therefore see any value in adding a JSON (Basic) which JSON is derived from by just adding some file extensions.

I'm still undecided if I should add more flavors in this step ... or do that later. This was just some preparation for (possibly) later.

JSON/JSON (Basic).sublime-syntax

JSON/Default.sublime-keymap

deathaxe · 2021-10-21T17:01:55Z

I'm still undecided if I should add more flavors in this step ... or do that later. This was just some preparation for (possibly) later.

Haven't investigated any other flavours nor being aware of them, but does that change anything?

The default JSON.sublime-syntax could still be the base for everything with its source.json main scope, no? It shouldn't even hurt to add the .basic and keep it as is.

So we'd end up in:

file	scope
JSON.sublime-syntax	`source.json.basic`
JSONC.sublime-syntax	`source.json.jsonc`
JSON (<flavor>).sublime-syntax	`source.json.<flavor>`

jrappen · 2021-10-21T17:21:01Z

Probably not. Unless I have to extract bits from JSON (Basic) to JSON. It's easier to merge than separate them later.

michaelblyons · 2021-10-21T19:13:46Z

It shouldn't even hurt to add the .basic and keep it as is.

What happens when another syntax - include: scope:source.json if you change the root scope to source.json.basic? Will something be included? If so, what?

jrappen · 2021-10-21T20:04:03Z

If so, what?

In that case I'd assume nothing.

deathaxe · 2021-10-21T20:10:11Z

You are right @michaelblyons. Nothing is included then. So it's not an option.

JSON/JSON (Basic).sublime-syntax

jrappen · 2021-10-21T20:15:41Z

Could use this instead, though:

- include: scope:Packages/JSON/JSON.sublime-syntax

unless I'm mistaken. I have a vague memory seeing this somewhere.

michaelblyons · 2021-10-21T21:30:42Z

Could use this instead, though:
- include: scope:Packages/JSON/JSON.sublime-syntax
unless I'm mistaken. I have a vague memory seeing this somewhere.

I know you can do this within a package. I believe Git Formats does so. You may be able to do it between packages (e.g. Markdown front matter).

But switching away from source.json would still break other people's packages all over the internet.

jrappen · 2021-10-21T21:32:18Z

Sometimes, breaking stuff is the way forward. No need for that here, though.

jrappen · 2021-10-21T21:38:45Z

On the other hand, the package you linked from across the internet I will definitely break, cause he'd have to switch to scope:source.json.jsonc#prototype as JSON won't have that (i.e. comments) anymore.

jrappen · 2021-10-21T22:07:57Z

Does anybody know of any JSON flavors that do accept decimal floats with leading or trailing periods in the base?

In basic JSON they apparently are illegal (and weren't scoped as such, yet).

jrappen · 2021-10-21T22:24:59Z

⚠️ I have addressed and resolved all comments above, except for the one about meta_include_prototype.

Decimal floats are now forced to a more explicit format.

jrappen · 2021-10-21T23:19:38Z

Thoughts on embed: scope:source.json in:

LaTeX
Markdown
Perl
PHP Source

and push: scope:source.json in:

PHP Source

anyone?

jrappen · 2021-11-06T17:02:23Z

I personally believe that if someone chooses JSON as syntax, it should be highlighted as such. The current situation does not allow this using a default installation of Sublime Text. This was the starting point for me as well as FichteFoll in the issue linked in the opening post.

Deviating from the current situation and introducing other flavors of JSON will cause breaking changes. Either one way or another.

We can either (1) keep JSONC be a JSON syntax and introduce a JSON Basic (to make inheritance work) and JSON Strict (as a true JSON syntax). Any monkey patching around wrongly labeled syntaxes we avoid now in core and for any third parties will simply be defered to the future where third parties will have to address JSON Strict instead of JSON. Of course they will figure it out, but maybe not right away. Maybe this is less overall cost of time. Maybe not. I see this as offloading past problems to others in the future. And continuing to do so, as this has been an issue for a while and got us here in the first place. We can agree to disagree on this, I'm just stating my point.

Or we can (2) cause a bit of a stir now but be at peace for a long while afterwards. Hence why we can test this in the dev release channel for a while, before this hits stable. I'd assume package authors of frequently used packages are on the dev channel anyway.

We can also wait with this, until there is a longer cycle, if now is not a good time.

Not sure why you need to stress that the status bar label for the JSONC syntax is so ugly currently. I already agreed to change it to just JSONC. You can auto hide it while typing if desired. No big deal.

The JSON syntax exists for files with patterns listed in file_extensions, but you knew that. Why are you asking? If we remove the JSON syntax, what is the remaining point of this PR? Just adding JSON5?

If the bright red colors are such a bother, we have a few options. We can (1) either keep it and you add an override to Mariana to dim your reds, or (2) SublimeHQ will do it for you (which will never happen as we both know) or I (3) remove all illegal highlighting from JSON and add a linter that runs on the json module which ships with the Python included with Sublime and we have underlines or outlines - or whatever choice we agree upon - to highlight invalid regions (in this case). Or (4) I just remove all illegal highlighting, but then there really is no big difference between the highlighting of JSON vs JSONC.

Color schmes should follow syntaxes and (personally) I don't mind the red highlight of (temporarily or not but nonetheless) illegal code.

Because we are discussing:

trailing periods within floats
and trailing commas within structures (objects and arrays)

the user experience is flashing if we assume the user types fast. For you the flashing is more important (and disturbing) than is the highlighting of illegal code for me. We can again agree to disagree here, perfectly fine.

The problem for me as an author of the syntax file is though, that I cannot differentiate between "user typing" and "user done typing". Unless there is a way of "defering the highlight until after typing".

My suggestion is having a separate JSON syntax and writing it as it should be, not as the current user with old habits is used to. We can again agree to disagree.

If most users agree with youk then we should agree to close this PR as it will not benefit you or them. Hence my suggestion.

Which is fine with me, I can use this locally. I don't see this as having wasted my time. I have personal use for this.

keith-hall · 2021-11-06T19:43:50Z

My 2 cents on this one issue about trailing commas being highlighted as invalid (while typing/not typing - we have no distinction as has been said already):

They are illegal in "strict JSON", and should be highlighted as such. Yes, it is possible that it could be annoying for users while typing. But:

the benefits outweigh the drawbacks - it will likely save lots of trouble with people new to JSON who don't realize that trailing commas are illegal. "ST allows it for it's settings, but it's not working in my [language X] application. hmm... ah, I see that the syntax is different - JSONC vs JSON. Probably that's got something to do with it..."
if we scope multiple consecutive commas distinctly from one trailing comma, color schemes/users can choose the behavior they want for trailing commas. ST isn't a "one size fits all" out of the box - it's designed to be customized.
users can use snippets etc. to minimize the "flashes of red" by avoiding having trailing commas while typing at all
how many people actually write JSON (and I mean "JSON (strict)" here I guess ;)) by hand these days?

jrappen · 2021-11-06T19:52:08Z

I would agree with you, Janos, if this was a constant flicker like if you have two surfaces in the same plane in 3D and your maschine is constantly trying to render both. This is a single flash though of one single character which goes away after typing one (within the float) or two (within an array or object) additional characters.

For this reason I find this acceptable.

jrappen · 2021-11-17T19:45:06Z

Hello guys, I didn't forget this. Simply didn't have the time to take care of this during the last couple days. I'll update this towards the weekend, as time permits.

jrappen · 2021-12-02T23:01:02Z

⚠️ Status update

Some TODOs remaining before I'll open this for review:

JSON object: leading separators
JSON object: trailing commas
JSON numbers: decimal floats with leading and trailing periods
JSON5: valid whitespace

keith-hall · 2021-12-03T19:37:31Z

I was going to ask whether you plan to work on scoping for dates, but then realized that what I've seen in the wild is not standard in any official JSON variant: https://davidsekar.com/javascript/converting-json-date-string-date-to-date-object

michaelblyons · 2021-12-03T20:04:03Z

I was going to ask whether you plan to work on scoping for dates, but then realized that what I've seen in the wild is not standard in any official JSON variant: https://davidsekar.com/javascript/converting-json-date-string-date-to-date-object

Haha. The old Microsoft JSON date format, leading to one of my Stack Overflow bookmarks.

jrappen · 2021-12-04T08:53:07Z

I was going to ask whether you plan to work on scoping for dates ...

I could still add them, though. I believe it adds value?

JSON/JSON (Basic).sublime-syntax

jrappen · 2021-12-09T09:05:32Z

@keith-hall Could you paste me a few links with references for your last request? I'd appreciate it.

I won't be adding your request to the JSON flavors I currently have (JSON, JSONC, JSON5), but rather add another flavor. I believe it's less confusing to add another flavor than to add this kind of stuff to the ones we already have and where it's not expected.

Are we talking about this?

keith-hall · 2021-12-09T09:17:42Z

These links may be useful:
https://www.newtonsoft.com/json/help/html/DatesInJSON.htm#DatesAndJsonNET
https://www.developer.com/microsoft/dealing-with-json-dates-in-asp-net-mvc/
https://github.com/featurist/dotnet-json-date

deathaxe · 2021-12-12T16:53:02Z

The current implementation causes trouble with partial json in Markdown fenced code blocks.

Hence I am not convinced of adding too much of such over eager illegal syntax highlighting stuff.

jrappen · 2021-12-12T21:10:18Z

causes trouble with partial json in Markdown fenced code blocks

I've had this on my checklist for a while.

I initially thought it might make sense to have a (JSON) "snippet" syntax, since JSON is a comparatively simple syntax.

deathaxe · 2021-12-13T17:38:22Z

JSON snippet syntax? There are already enought dialects. Adding more doesn't make sense. I'd rather relax existing JSON to not complain about incomplete globals.

jrappen · 2022-04-05T22:21:16Z

Should I meta-scope links and email addresses in value strings as in the Markdown re-write?

Markdown/syntax_test_markdown.fenced_code_json.md

@deathaxe

- Using inheritance split up `JSON.sublime-syntax` into: - `JSON.sublime-syntax` with `scope:source.json` - `JSONC.sublime-syntax` with `scope:source.json.jsonc` - `JSON5.sublime-syntax` with `scope:source.json.json5` - `JSON_dotNET.sublime-syntax` with `scope:source.json.json-dotnet` - Add many more file extensions for `JSON` & `JSONC`: - add doc links to extensions where applicable as a reference to be able to more quickly verify that they (still) use said syntax flavor - JSON: - Make use of newer syntax features including those only available in `version: 2` syntaxes - Make use of `variables` (with optimizations provided by @deathaxe and regex patterns provided by @Thom1729) - Context names now more closely match the naming scheme of other (recently re-written) default syntaxes - (correctly formatted) JSON code can now be prettified or minified via the context menu or the command palette. JSON code can optionally be auto-prettified on pre save events. - highlight leading, trailing & multiple commas as invalid - only allow exactly one structure (object, array) or value (constant, number, string) at top level (thanks to @keith-hall) - links (`meta.link.inet`) and email addresses (`meta.link.email`) are scoped the same as in Markdown (thanks to @deathaxe) - JSONC: - highlight some files by default as `JSONC` (as decided by @jskinner in sublimehq#285) - highlight leading & multiple commas as invalid, trailing as valid - scope empty block comments as such - support syntax based folding of ST4131+, compare sublimehq#3291 - JSON5: - explicitly pos numbers, hexadecimal ints, Infinity and NaN - single quoted strings - more escape chars for strings - ECMA identifierName as object keys (regexes thanks to @Thom1729) - scoped as plain unquoted strings (thanks to @Thom1729) - support string interpolation (thanks to @deathaxe) - line continuation in strings (with tests thanks to @keith-hall) - JSON.NET: - support requested by @keith-hall, built with feedback from @michaelblyons - Objects: - Highlighting speed improvements for empty objects (thanks to @FichteFoll) - Make `mapping.*` contexts more modular - Arrays: - Highlighting speed improvements for empty arrays (thanks to @FichteFoll) - Numbers: - Correctly scope number signs with `constant.numeric.sign` instead of `keyword.operator.arithmetic` - Significantly improve number highlighting (thanks to @deathaxe) - Completions: - completions have been added for language constants, including kind info and details (with links to docs) - `null`, `false`, `true` for JSON - `Infinity` and `NaN` for JSON5 - Settings: - a `default_extension` is now set for all JSON flavors - Symbol index: - with an object structure at the top-level, only top-level keys within now show up in the index (including tests for symbols and syntax) - Tests: - test files now test the base scope - Significantly extend tests to cover more parts of the syntaxes - Split original test file into logical parts - Add indentation tests for: - `json`, `json5` & `jsonc` - `mapping` (objects), `sequence` (arrays) - Add symbols tests for: - top-level keys of object structures (thanks to deathaxe) - languages: `json`, `json5` & `jsonc` - Fix tests for `meta.mapping meta.mapping.*` - Leave `JSON` headers in `Markdown` as `json` only, but split up fenced code blocks into `json`, `json5` & `jsonc` to behave similarly to `GitHub Flavored Markdown` BREAKING CHANGES: - JSON does not have values that can be set via an inline calculation with the help of operators, but only simple number values. Scopes for number signs have changed from being `keyword.operator.arithmetic` to `constant.numeric.sign`. Color scheme authors should add this, should it be missing. - The `JSON.sublime-syntax` now marks comments as `invalid`, third party plugin authors should instead target `JSONC.sublime-syntax` to keep the user experience as-is. - Indexed symbols (i.e. top-level keys in JSON object structures) are scoped as `source.json meta.mapping.key - (meta.mapping.value meta.mapping.key | meta.sequence.list meta.mapping.key)`. Color scheme authors should add special highlighting to differentiate them from other keys. - fix sublimehq#285 - address sublimehq#421 (thanks to @FichteFoll) - address sublimehq#481 to remove incompatible regex patterns according to @wbond - address sublimehq#757 to fix line comments for `JSONC` (thanks to @keith-hall) - address sublimehq#2430 using sort-order (as requested by @deathaxe) - address sublimehq#2711 with regards to `constant.language.null` vs. `constant.language.empty` (thanks to @FichteFoll) - address sublimehq#2852 to fix scopes of curly braces & square brackets in `JSON` (thanks to @Thom1729) - address sublimehq#3228 to fix `punctuation.separator` scopes, compare sublimehq#3270 - address sublimehq/sublime_text#3154 and add symbol tests Co-authored-by: Ashwin Shenoy <Ultra-Instinct-05@users.noreply.github.com> Co-authored-by: Jack Cherng <jfcherng@users.noreply.github.com> Co-authored-by: Janos Wortmann <jwortmann@users.noreply.github.com> Co-authored-by: Jon Skinner <jps@sublimetext.com> Co-authored-by: FichteFoll <FichteFoll@users.noreply.github.com> Co-authored-by: Keith Hall <keith-hall@users.noreply.github.com> Co-authored-by: Michael B. Lyons <michaelblyons@users.noreply.github.com> Co-authored-by: Rafał Chłodnicki <rchl@users.noreply.github.com> Co-authored-by: Thomas Smith <Thom1729@users.noreply.github.com> Co-authored-by: Will Bond <wbond@users.noreply.github.com> Co-authored-by: deathaxe <deathaxe@users.noreply.github.com>

michaelblyons · 2022-06-01T01:00:37Z

Just so you know, the @ mention in the commit description means we're getting an email for every force push.

jrappen · 2022-06-01T01:16:18Z

Sure, but if I use bold text, it wont be linked up correctly after merge.

Ultra-Instinct-05 · 2022-06-01T10:39:52Z

I am not really sure if it's good idea to ship minification/prettification plugins for JSON with the Default package. Would be a really nice excuse for people to complain about why other syntax packages don't ship with such. IMHO, best suited for a 3rd party package. Maybe LSP-json even has such capabilities, but haven't checked.

TerminalFi · 2023-02-27T21:24:35Z

Not a fan of built in minify and prettify.It is adding another entry point for UI sluggishness. I imagine this being shipped and then issues being filed for Sublime being slow when prettifying or minifying json.

This is better suited for a plug-in.

jrappen · 2023-02-27T21:37:07Z

I'll update the PR when I have some more time later this week ... or so.

jrappen marked this pull request as draft October 21, 2021 00:18

jrappen force-pushed the fix-json branch from 0031678 to 7ddeece Compare October 21, 2021 00:55