Semantic Tokens support

[BoykoAlex]

These will be rather questions than issues @rubenporras

There is no support for workspace/semanticTokens/refresh message. At first I thought it is easy to add. I've tried the following the LangugeClientImpl:

	private void updateSemanticTokens() {
		for (IWorkbenchWindow window : PlatformUI.getWorkbench().getWorkbenchWindows()) {
			for (IWorkbenchPage page : window.getPages()) {
				for (IEditorReference editorRef : page.getEditorReferences()) {
					IEditorPart editor = editorRef.getEditor(false);
					if (editor != null) {
						ITextViewer textViewer = UI.asTextViewer(editor);
						if (textViewer != null) {
							textViewer.invalidateTextPresentation();
						}
					}
				}
			}
		}
	}

	@Override
	public CompletableFuture<Void> refreshSemanticTokens() {
		return CompletableFuture.runAsync(() -> UI.getDisplay().asyncExec(this::updateSemanticTokens));
	}

Didn't work :-\ Any idea what I can do to make SemanticHighlightReconcilerStrategy call reconcile(final DirtyRegion dirtyRegion, final IRegion subRegion) ?

Another thing that I noticed is only keyword and variable token types had colours in the Generic editor (I'm trying to do some semantic highlighting in the properties file). All other token types listed in the LSP spec don't have any colour assigned. Is there a way for me to define colours for other token types?

[rubenporras]

@BoykoAlex ,

• invalidateTextPresentation will not refresh the semantic tokens, only apply again the current ones to the text presentation. With the current code we would need to call SemanticHighlightReconcilerStrategy.initialReconcile, and it looks like to do that one would need to uninstall/install the reconciler. I have not found any API to do that from the sourceViewer.
  I wonder if what would need is a new entry in org.eclipse.jface.text.source.SourceViewer.enableOperation(int, boolean) so that we could programmatically uninstall/install the reconciler.
  How does that sound to you? I think another way would be to react to some system property or any other global configuration in SemanticHighlightReconcilerStrategy.applyTextPresentation to also trigger the refresh of the semantic tokens, but that seems less clean to me.

• Regarding "only keyword and variable token types had colours in the Generic editor", I think that is because we derive the style for the tokens from the TexMate stylesheet, and the keyword and variable will be defined in there. You can add other tokens even if they are not used by TexMate grammar, and then I believe our semantic highlight will use them.

[BoykoAlex]

@BoykoAlex ,

• invalidateTextPresentation will not refresh the semantic tokens, only apply again the current ones to the text presentation. With the current code we would need to call SemanticHighlightReconcilerStrategy.initialReconcile, and it looks like to do that one would need to uninstall/install the reconciler. I have not found any API to do that from the sourceViewer.
  I wonder if what would need is a new entry in org.eclipse.jface.text.source.SourceViewer.enableOperation(int, boolean) so that we could programmatically uninstall/install the reconciler.
  How does that sound to you? I think another way would be to react to some system property or any other global configuration in SemanticHighlightReconcilerStrategy.applyTextPresentation to also trigger the refresh of the semantic tokens, but that seems less clean to me.

Thus I'd be able to call something like sourceViewer.doOperation(ISourceViewer.RESET_PRESENTATION_RECONCILER)?
I think that would be great!

• Regarding "only keyword and variable token types had colours in the Generic editor", I think that is because we derive the style for the tokens from the TexMate stylesheet, and the keyword and variable will be defined in there. You can add other tokens even if they are not used by TexMate grammar, and then I believe our semantic highlight will use them.

I found keyword and variable in the CSS file in the source code for TM4E... However, I have no idea how I can add additional styles... I'd be happy to extra styles via extension point or preferences or Java API (if there is one). Any hint here would be much appreciated.

[rubenporras]

Hi @BoykoAlex ,

I had in mind two calls in a row:

sourceViewer.enableOperation(ISourceViewer.PRESENTATION_RECONCILER, false);
sourceViewer.enableOperation(ISourceViewer.PRESENTATION_RECONCILER, true)

Does it look reasonable? If so, would you mind creating the PR for the platform?

Regarding the styles, you can copy the contents of stylesheet you are using, add new entries and add your new stylesheet, something like:

   <extension
        point="org.eclipse.tm4e.ui.themes">
        <theme
              id="abc"
              name="My Light Theme"
              path="./themes/light.css"
              default="true" >
        </theme>
        <theme
              id="abcDark"
              name="My Dark Theme"
              path="./themes/dark.css"
              default="true"
              dark="true" >
        </theme>
   </extension>