Suggestions widget doesn't open inside content tokenized with the name 'string'

[cdussud]

Completion suggestions don't automatically show up between curly braces while editing Markdown in Monaco.

After playing with the Monarch definition for Markdown it turns out that the token for { } is called 'string.target'. Any token containing the word 'string' suppresses suggestions, and that renaming that token to something else will allow suggestions to work normally.

This has to be intentional, so my real question is: is this behavior configurable? Can I turn this off? Or do I have to effectively rename the token to get the desired behavior?

monaco-editor version: 0.20.0
Browser: Chrome
OS: Mac OS
Playground code that reproduces the issue:

monaco.languages.registerCompletionItemProvider('markdown', {
    provideCompletionItems: function() {
        return {
            suggestions: [
                {
                    label: "random_number",
                    insertText: "random_number"
                },
                {
                    label: "now",
                    insertText: "now"
                },
                {
                    label: "today",
                    insertText: "today"
                },
            ]
        };
    }
});

monaco.editor.create(document.getElementById("container"), {
    value: "# hello.\n{autocomplete doesn't work between braces: }\nbut it works outside: ",
    language: "markdown"
});

Playground code that shows a workaround:

This example makes a new language using a copy of the Markdown definition file https://github.com/microsoft/monaco-languages/blob/master/src/markdown/markdown.ts#L143

I renamed the token matching { } to curly-brace from string.target. With that change autocomplete works inside the braces now.

// Register a new language
monaco.languages.register({ id: 'mySpecialLanguage' });

// Register a tokens provider for the language
monaco.languages.setMonarchTokensProvider('mySpecialLanguage', {
    defaultToken: '',
    tokenPostfix: '.md',

    // escape codes
    control: /[\\`*_\[\]{}()#+\-\.!]/,
    noncontrol: /[^\\`*_\[\]{}()#+\-\.!]/,
    escapes: /\\(?:@control)/,

    // escape codes for javascript/CSS strings
    jsescapes: /\\(?:[btnfr\\"']|[0-7][0-7]?|[0-3][0-7]{2})/,

    // non matched elements
    empty: [
        'area', 'base', 'basefont', 'br', 'col', 'frame',
        'hr', 'img', 'input', 'isindex', 'link', 'meta', 'param'
    ],
    tokenizer: {
        root: [

            // markdown tables
            [/^\s*\|/, '@rematch', '@table_header'],

            // headers (with #)
            [/^(\s{0,3})(#+)((?:[^\\#]|@escapes)+)((?:#+)?)/, ['white', 'keyword', 'keyword', 'keyword']],

            // headers (with =)
            [/^\s*(=+|\-+)\s*$/, 'keyword'],

            // headers (with ***)
            [/^\s*((\*[ ]?)+)\s*$/, 'meta.separator'],

            // quote
            [/^\s*>+/, 'comment'],

            // list (starting with * or number)
            [/^\s*([\*\-+:]|\d+\.)\s/, 'keyword'],

            // code block (4 spaces indent)
            [/^(\t|[ ]{4})[^ ].*$/, 'string'],

            // code block (3 tilde)
            [/^\s*~~~\s*((?:\w|[\/\-#])+)?\s*$/, { token: 'string', next: '@codeblock' }],

            // github style code blocks (with backticks and language)
            [/^\s*```\s*((?:\w|[\/\-#])+).*$/, { token: 'string', next: '@codeblockgh', nextEmbedded: '$1' }],

            // github style code blocks (with backticks but no language)
            [/^\s*```\s*$/, { token: 'string', next: '@codeblock' }],

            // markup within lines
            { include: '@linecontent' },
        ],

        table_header: [
            { include: '@table_common' },
            [/[^\|]+/, 'keyword.table.header'], // table header
        ],

        table_body: [
            { include: '@table_common' },
            { include: '@linecontent' },
        ],

        table_common: [
            [/\s*[\-:]+\s*/, { token: 'keyword', switchTo: 'table_body' }], // header-divider
            [/^\s*\|/, 'keyword.table.left'], // opening |
            [/^\s*[^\|]/, '@rematch', '@pop'], // exiting
            [/^\s*$/, '@rematch', '@pop'], // exiting
            [/\|/, {
                cases: {
                    '@eos': 'keyword.table.right', // closing |
                    '@default': 'keyword.table.middle', // inner |
                }
            }],
        ],

        codeblock: [
            [/^\s*~~~\s*$/, { token: 'string', next: '@pop' }],
            [/^\s*```\s*$/, { token: 'string', next: '@pop' }],
            [/.*$/, 'variable.source'],
        ],

        // github style code blocks
        codeblockgh: [
            [/```\s*$/, { token: 'variable.source', next: '@pop', nextEmbedded: '@pop' }],
            [/[^`]+/, 'variable.source'],
        ],

        linecontent: [

            // escapes
            [/&\w+;/, 'string.escape'],
            [/@escapes/, 'escape'],

            // various markup
            [/\b__([^\\_]|@escapes|_(?!_))+__\b/, 'strong'],
            [/\*\*([^\\*]|@escapes|\*(?!\*))+\*\*/, 'strong'],
            [/\b_[^_]+_\b/, 'emphasis'],
            [/\*([^\\*]|@escapes)+\*/, 'emphasis'],
            [/`([^\\`]|@escapes)+`/, 'variable'],

            // links
            [/\{+[^}]+\}+/, 'curly-brace'],
            [/(!?\[)((?:[^\]\\]|@escapes)*)(\]\([^\)]+\))/, ['string.link', '', 'string.link']],
            [/(!?\[)((?:[^\]\\]|@escapes)*)(\])/, 'string.link'],

            // or html
            { include: 'html' },
        ],

        // Note: it is tempting to rather switch to the real HTML mode instead of building our own here
        // but currently there is a limitation in Monarch that prevents us from doing it: The opening
        // '<' would start the HTML mode, however there is no way to jump 1 character back to let the
        // HTML mode also tokenize the opening angle bracket. Thus, even though we could jump to HTML,
        // we cannot correctly tokenize it in that mode yet.
        html: [
            // html tags
            [/<(\w+)\/>/, 'tag'],
            [/<(\w+)/, {
                cases: {
                    '@empty': { token: 'tag', next: '@tag.$1' },
                    '@default': { token: 'tag', next: '@tag.$1' }
                }
            }],
            [/<\/(\w+)\s*>/, { token: 'tag' }],

            [/<!--/, 'comment', '@comment']
        ],

        comment: [
            [/[^<\-]+/, 'comment.content'],
            [/-->/, 'comment', '@pop'],
            [/<!--/, 'comment.content.invalid'],
            [/[<\-]/, 'comment.content']
        ],

        // Almost full HTML tag matching, complete with embedded scripts & styles
        tag: [
            [/[ \t\r\n]+/, 'white'],
            [/(type)(\s*=\s*)(")([^"]+)(")/, ['attribute.name.html', 'delimiter.html', 'string.html',
                { token: 'string.html', switchTo: '@tag.$S2.$4' },
                'string.html']],
            [/(type)(\s*=\s*)(')([^']+)(')/, ['attribute.name.html', 'delimiter.html', 'string.html',
                { token: 'string.html', switchTo: '@tag.$S2.$4' },
                'string.html']],
            [/(\w+)(\s*=\s*)("[^"]*"|'[^']*')/, ['attribute.name.html', 'delimiter.html', 'string.html']],
            [/\w+/, 'attribute.name.html'],
            [/\/>/, 'tag', '@pop'],
            [/>/, {
                cases: {
                    '$S2==style': { token: 'tag', switchTo: 'embeddedStyle', nextEmbedded: 'text/css' },
                    '$S2==script': {
                        cases: {
                            '$S3': { token: 'tag', switchTo: 'embeddedScript', nextEmbedded: '$S3' },
                            '@default': { token: 'tag', switchTo: 'embeddedScript', nextEmbedded: 'text/javascript' }
                        }
                    },
                    '@default': { token: 'tag', next: '@pop' }
                }
            }],
        ],

        embeddedStyle: [
            [/[^<]+/, ''],
            [/<\/style\s*>/, { token: '@rematch', next: '@pop', nextEmbedded: '@pop' }],
            [/</, '']
        ],

        embeddedScript: [
            [/[^<]+/, ''],
            [/<\/script\s*>/, { token: '@rematch', next: '@pop', nextEmbedded: '@pop' }],
            [/</, '']
        ],
    }
});

// Define a new theme that contains only rules that match this language
monaco.editor.defineTheme('myCoolTheme', {
    base: 'vs',
    inherit: false,
    rules: [
        { token: 'keyword', foreground: '808080' },
        { token: 'custom-error', foreground: 'ff0000', fontStyle: 'bold' },
        { token: 'custom-notice', foreground: 'FFA500' },
        { token: 'string', foreground: '882200' },
        { token: 'curly-brace', foreground: 'abcdef'}
    ]
});

// Register a completion item provider for the new language
monaco.languages.registerCompletionItemProvider('mySpecialLanguage', {
    provideCompletionItems: () => {
        return {
            suggestions: [
                {
                    label: "random_number",
                    insertText: "random_number"
                },
                {
                    label: "now",
                    insertText: "now"
                },
                {
                    label: "today",
                    insertText: "today"
                },
            ]
        };
    }
});

monaco.editor.create(document.getElementById("container"), {
    theme: 'myCoolTheme',
    value: getCode(),
    language: 'mySpecialLanguage'
});

function getCode() {
    return "# hello"
}

[alexdima]

PR welcome. I don't know why this rule exists: [/\{+[^}]+\}+/, 'string.target']. The only hint is a comment above it: 'links'.

[mewalig]

The issue does not seem to be related to the name of the target. If you take your code, put it into Monaco playground, and change 'curly-brace' back to string.target, it also works.

Probably, it has to do with how monaco filters the current text against the token text. when your suggestions return "today", monaco is not matching because the token starts with "{" but the suggestion does not. Modification using filterText and range can address this issue, though I'm not sure why monaco sometimes seems to care about the token starting with a bracket and other times does not.

UPDATE: on further review it seems to work fine with your originally-posted code. Perhaps you are expecting it to auto-suggest based on the current word, if one were to break up the text within the curly braces into separate words? Keep in mind that {...} is a single token, so autocomplete will try to match the prefix consisting of all of the text starting with the first char after the bracket until the cursor position-- i.e. "{hello there" and not just "there":

So for example:

Which is from your original code, with one additional suggestion:

monaco.languages.registerCompletionItemProvider('markdown', {
    provideCompletionItems: function() {
        return {
            suggestions: [
                {
                    label: "random_number",
                    insertText: "random_number"
                },
                {
                    label: "now",
                    insertText: "now"
                },
                {
                    label: "today",
                    insertText: "today"
                },
                {
                    label: "autocomplete does work fine between braces",
                    insertText: "autocomplete does work fine between braces"
                },
            ]
        };
    }
});

monaco.editor.create(document.getElementById("container"), {
    value: "# hello.\n{autocomplete doesn't work between braces: }\nbut it works outside: ",
    language: "markdown"
});