Major Changes for Downstream

[alexr00]

It looks like new used to be keyword.control.cpp. Now it is keyword.operator.memory.new.cpp. Is this intentional? If it's intentional, we can add the it into the themes in VS Code, but ideally there wouldn't be too many changes like this since it requires constant updating of the themes.

[alexr00]

Original issue: microsoft/vscode#71821

[jeff-hykin]

Yes it was intentional, and I agree we will try to keep changes like this to a minimum since many themes already setup with the old way.

From here on out, I'll put any similar changes on this post.

C++ Log as of August 12th 2019

• Helper-tags have been added to make tag changes much easier to maintain. None of these change/replace existing tags
  • (1.7.0) keyword.operator.functionlike.cpp keyword.operator
    will match all of the operators like alignas, sizeof, etc that require parentheses
  • (1.7.0) keyword.operator.wordlike.cpp keyword.operator
    will match all of the operators that use alpha numeric characters (that are not functionlike)
  • (1.7.0) memory keyword.operator
    will match both keyword.operator.new and keyword.operator.delete
• new and delete changed
  • (1.0.0) keyword.control
  • (1.4.0) keyword.operator.memory.new
  • (1.7.0) keyword.operator.new and keyword.operator.delete
• The using keyword changed
  • (1.0.0) keyword.control
  • (1.4.0) keyword.other.using
• and, or, xor, etc have received additional specificity
  • (1.7.0) match them with alias keyword.operator
  • (1.11.7) match them with keyword.operator.wordlike keyword.operator or keyword.operator.or
• The operator keyword was changed
  • (1.0.0) keyword.control
  • (1.5.0) keyword.other.operator
• Member access received additional scopes
  • (1.4.8) variable.other.object.accessor
    • To effectively revert this set the color of this tag to be the same as source.cpp
• Member variables have changed
  • (1.0.0) variable.other.member
  • (1.4.8) variable.other.object.property (middle) and variable.other.property (end)
    • this was done to better accommodate Local variables are randomly highlighted incorrectly. #59 and be consistent across languages (see Java, Javascript, C#, Typescript, CoffeeScript)
• The namespace {}'s have changed
  • (1.0.0) punctuation.definition.scope
  • (1.7.0) punctuation.section.block.begin.bracket.curly.namespace.cpp punctuation.section.block.end.bracket.curly.namespace.cpp
    • This change is because class/struct/enum/union/extern all use that for their brackets. Namespace brackets are the only anti-pattern
• decltype is now tagged
  • (1.6.0) both storage.type.decltype and keyword.other.decltype
• The class, struct, union, enum, extern, namespace pattern(s) were overhauled and standardized
  • (1.0.0) meta.class-struct-block, meta.extern-block, meta.namespace-block
  • (1.7.0) changed to
    • meta.block.class
    • meta.block.struct
    • meta.block.union
    • meta.block.enum
    • meta.block.extern
    • meta.block.namespace
• scope of namespaces has changed
  • (1.8.2) entity.name.type.namespace
  • (1.9.1) entity.name.namespace
• pthread types have changed
  • (1.0) support.type.pthread
  • (1.8.5) support.type.posix-reserved.pthread
• change to ()'s
  • (1.0) punctuation.section.parens
  • (1.8.6) meta.parens.c
  • [ subject to change in future, it will stay meta, but might be meta.block or similar ]
• function pointers
  • (1.0) no tagging
  • (1.8.14) experimental tagging inside of structs/classes only
  • (1.9.0) tagging inside of function parameters
• pointer and reference
  • (1.0) keyword.operator
  • (1.8.14) storage.modifier.pointer and storage.modifier.reference
• scope resolution identifiers
  • (1.0?) entity.name.type.namespace.scope-resolution
  • (1.9.0)
    • entity.name.scope-resolution
    • entity.name.scope-resolution.function.call
    • entity.name.scope-resolution.function.definition
    • entity.name.scope-resolution.function.definition.operator-overload
    • entity.name.scope-resolution.namespace.alias
    • entity.name.scope-resolution.namespace.block
    • entity.name.scope-resolution.namespace.using
    • entity.name.scope-resolution.template.call
    • entity.name.scope-resolution.template.definition
• scope resolution punctuation ::
  • (1.0) punctuation.separator.namespace.access
  • (1.9.0)
    • punctuation.separator.scope-resolution
    • punctuation.separator.scope-resolution.function.call
    • punctuation.separator.scope-resolution.function.definition
    • punctuation.separator.scope-resolution.function.definition.operator-overload
    • punctuation.separator.scope-resolution.namespace.alias
    • punctuation.separator.scope-resolution.namespace.block
    • punctuation.separator.scope-resolution.namespace.using
    • punctuation.separator.scope-resolution.template.call
    • punctuation.separator.scope-resolution.template.definition
  • (1.9.2) planned to add punctuation.separator.namespace.access for better backwards compatibility
• template call angle brackets
  • (1.0) keyword.operator.comparision
  • (1.9.0) more of them are now tagged as
    • punctuation.section.angle-brackets.begin.template.call
    • punctuation.section.angle-brackets.end.template.call
    • punctuation.section.angle-brackets.begin.template.definition
    • punctuation.section.angle-brackets.end.template.definition
• macro parentheses
  • (1.0) punctuation.section.arguments.begin.bracket.round.cpp
  • (1.0) punctuation.section.arguments.end.bracket.round.cpp
  • (1.13.0) punctuation.definition.parameters.begin.preprocessor
  • (1.13.0) punctuation.definition.parameters.end.preprocessor

[jeff-hykin]

The reason it was changed is because new and delete are completely unrelated to control flow. I picked operator since that's what what I believed it was classified under the C++ spec.

@matter123 pointed out in his pull request its only an operator under a special case. Additionally its debatable that the coloration would be improved if it were labeled with keyword.other.

@alexr00 should we go ahead and make the change from keyword.operator to keyword.other?

[alexr00]

Having the big changes here is perfect, thank you. I took a quick look at how other languages handle new:
coffee
keyword.operator.new.coffee

JS
keyword.operator.new.js

groovy
keyword.control.new.groovy

java
keyword.control.new.java

There isn't a real consensus. Based on what the VS Code themes already have, keyword.operator.new would be good. keyword.other, like keyword.operator.memory would still require changes to the theme. I'm happy to make minor updates to themes though. If you prefer keyword.operator.memory more, then feel free to leave it and I'll make the update to include it here and in the equivalent places in the other themes.

[matter123]

keyword.operator.new is probably the best. I noticed on [expr.new]/4 the standard refers to the new operator. Though change is still needed for keyword.operator.delete

[jeff-hykin]

Thats interesting @alexr00 I'll try to get a complete list of changes by the end of tonight. There is a file in this repo syntax_examples.cpp it is designed to visually debug the syntax highlighter changes. You can probably use it to quickly see how everything looks.

[jeff-hykin]

@alexr00 The list isn't 100% complete, but I believe it covers everything that would be considered an issue. Many of the changes were made tonight to try to get all of the changes done now. I probably won't be able to make changes for about a week or more, but I'll be around daily for GitHub communication.

Do note that these changes are C++ exclusive. We are working on an ideal way to port them to C.

I believe the new keyword.operator.wordlike, keyword.operator.functionlike should take care of most of the issues.

[jeff-hykin]

Today I added a tags.txt, that gets auto generated with the syntax. It just sorts and lists out all of the tags that are in the grammar, it should be useful for keeping track of when things are added and removed. @matter123

[jeff-hykin]

@alexr00 The Objective-C and Objective-C++ are fully restored/fixed.
Yesterday we expanded the repo's structure to accommodate multiple languages. We added both the Objective-C and Objective-C++ syntaxes along with integration tests for both of them. Neither of them are dependent on C and C++ repos anymore. There's also been some work done on improving the Objective-C++ syntax, but to be on the safe side those changes are not included.

Other Info About the Problem

If you want to know more about why this happened in the first place, see here: #107 (comment)

Once the updated Objective-C and Objective-C++ are in VS Code, we can actually instantly fix some long standing bugs related to atom/language-c#146

Scope Change Guide

To make pulling the latest changes easy, here's a scope-change guide for the "dark+" and "dark" themes.

Making a pattern specific to C++

I'd recommend adding
source.cpp keyword.operator.new
rather than
keyword.operator.new.cpp
The second way makes it where additional specificity like keyword.operator.new.overload.cpp will break the theme

Might want to change "Control flow keywords"

The only scope in the "Control flow keywords" that is control flow is keyword.control, all of the others (keyword.operator.new.cpp, keyword.operator.delete, keyword.other.using, keyword.other.operator) are unrelated to control flow. So you might want to rename that group to something like "Special Keywords"

Scope Names

dark plus
    // probably make these white #d4d4d4 (same as keyword.operator)
        "storage.modifier.pointer",
        "storage.modifier.reference"
    // probably put these with the "Control flow keywords"/"Special Keywords"
        "keyword.operator.functionlike keyword.operator", // typeid(), alignas(), etc
        "keyword.operator.wordlike keyword.operator",  // and, or, xor, not, etc
    // C++ attributes, maybe color them purple, blue, or light blue
        "entity.other.attribute",
        "punctuation.section.attribute.begin",
        "punctuation.section.attribute.end",
    // might want to put the overloadee in "Function declarations"
        "entity.name.operator.overloadee"
    // might want to add this to "Types declaration and references"
    // since its almost always a class or namespace that is the scope-resolution
        "entity.name.scope-resolution"

dark theme:
    // probably want to color blue:
        "support.type"
        "keyword.operator.wordlike keyword.operator"
    // probably make these white #d4d4d4 (same as keyword.operator)
        "storage.modifier.pointer"
        "storage.modifier.reference"

[alexr00]

@jeff-hykin thank you for the detailed description of the issue and for all the fixes and suggestions! We are planning on releasing soon, and I want all these changes to be in insiders for a while before they go out in stable. As soon as we have branched for release (probably in the next couple week days) I'll bring in your Objective C and Objective C++ changes.

[jeff-hykin]

Great 👍 and I agree, I'd be more comfortable with them staying in the insiders branch for awhile.

[jeff-hykin]

Starting at v1.22.0 you might get downstream complaints about no SQL highlighting.

• v1.22.0 removes the source.sql since it creates frequent unfixable cascading breakages. The side effect is that no SQL keywords will be highlighted inside of raw strings.
• Highlighting SQL is still possible, as raw SQL strings are given the string.quoted.double.raw.sql and/or string.quoted.double.raw.SQL textmate scope. Extensions/editors can target that scope to provide SQL highlighting for those that still want it. Not great I know, but I think its better than defaulting to cascading breakages.