Semantic highlighting only works for the last line

[luan-xiaokun]

I am trying to implement a semantic token provider for my language, here is a strange behaviour I got.

I am expecting that both myint and mybool are highlighted, but when I finish typing the second typedef, myint get unhighlighted. After commenting the second line, myint get re-highlighted. Other tests and this suggest that only the last line gets correctly highlighted. Also, you can observe that when I start typing bool, it first gets highlighted as an alias, but finally gets highlighted as a keyword.

A minimal reproducible example is available here, and the langium I use is 0.4.0. A snippet of the syntax and the semantic token provider is as follows:

grammar Mediator

entry Program:
    TypeDefinition*;

TypeDefinition infers TypeDefinition:
    'typedef' name=TypeSpecifier 'as' alias=ID ';';

TypeSpecifier:
    (
        'int' | 'real' | 'char' | 'bool'
    )
    ('[' Expression? ']')?
    ('init' init=Expression)?
;

Expression:
    Addition;

Addition infers Expression:
    Multiplication ({infer BinaryExpression.left=current} operator=('+' | '-') right=Multiplication)*;

Multiplication infers Expression:
    PrimaryExpression ({infer BinaryExpression.left=current} operator=('*' | '/') right=PrimaryExpression)*;

PrimaryExpression infers Expression:
    '(' Expression ')' |
    {infer NumberLiteral} value=(INT | FLOAT);


terminal ID returns string: /[a-zA-Z_]\w*/;
terminal FLOAT returns number: /\d+\.\d*/;
terminal INT returns number: /[1-9]\d*|0/;

hidden terminal WS: /\s+/;
hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

import { SemanticTokenTypes } from 'vscode-languageserver';
import { AbstractSemanticTokenProvider, AstNode, SemanticTokenAcceptor } from 'langium';
import { isTypeDefinition } from './generated/ast';

export class MediatorGrammarSemanticTokenProvider extends AbstractSemanticTokenProvider {

    protected highlightElement(node: AstNode, acceptor: SemanticTokenAcceptor): void {
        if (isTypeDefinition(node)) {
            acceptor({
                node,
                feature: 'alias',
                type: SemanticTokenTypes.type
            });
        }
    }
}

Is there something wrong with my code? After comparing my code with the langium vscode extension's semantic token provider, I still cannot figure out why my implementation gets this strange behaviour while the langium vscode extension works well.

I have done a search for semantic highlighting discussions and issues but did not get very useful information, and the only reference I have is the langium vscode extension. Any reference to semantic highlighting implementation via langium would also be very helpful.

[msujew]

Hi @luan-xiaokun,

there's a minor issue in your grammar that prevents the semantic token provider from working correctly (and a lot of other services as well). Your AST model only consists of a single TypeDefinition, even though it parses multiple ones. You can easily fix this issue though:

entry Program:
    typeDefintions+=TypeDefinition*;

[luan-xiaokun]

Many thanks for your timely reply! It feels so good to fix this bug! Now I can continue to work on the semantic token provider itself.

[luan-xiaokun]

By the way, what is the correct way to highlight the content of an array type property (something like enumeration member)?

For example, here is the syntax for enumeration and corresponding semantic highlighting code. But this results in only the first enumeration member being highlighted.

EnumSpecifier:
    'enum' '{' (members+=ID (',' members+=ID)* ','?)? '}';

if (isEnumSpecifier(node)) {
    acceptor({
        node,
        feature: 'members',
        type: SemanticTokenTypes.enumMember
    });
}

[msujew]

It's not entirely correct. So by default, if you have an array of elements, you'll need to provide the index to the acceptor object. If you don't it will always take the first (i.e. an index of 0). You'll probably need to iterate from 0 to members.length and call the acceptor function. Though that makes me think that users probably want to highlight all elements of an array if omitting index. I'll probably change that.

[luan-xiaokun]

I see. Thanks a lot! Some comments for the SemanticTokenAcceptorOptions type in the source code would also be very helpful.