How to Handle Asynchronous Scoping

[ballcoach12]

I am working on a scope provider that will load an imported scope. It will do so by calling a remote endpoint via a network call and asking for the code in the imported package. It will then parse the package contents and provide the scope. The problem I'm seeing is that parsing is an asynchronous activity and the ScopeProvider interface expects a return type of Scope not Promise<Scope>. So I am not sure how to structure things so that I can obtain the AST for the imported package.

Is there a recommended way to do this?

[ballcoach12]

After looking at this a bit further, it appears that all I need to do is obtain the LangiumDocument , which can be parsed as shown in the following class method:

private getDocument(input: string, packageName:string): LangiumDocument<DtsPackage> {
        const uri = URI.parse(packageName);
        const result = this.services.shared.workspace.LangiumDocumentFactory.fromString<DtsPackage>(input, uri);
       return result;
    }

I assume that once I get this document, I can run validation on it to make sure it's valid.

[msujew]

Parsing is synchronous in Langium, but loading something from a remote resource likely isn't. How we usually do this (i.e. additional resources from something like a network resource), is by overriding the WorkspaceManager#loadAdditionalDocuments method and loading the resources as soon as we load our workspace. See also the documentation on built-in libraries.

[msujew]

Oh, yeah I always forget that update assumes those documents need to be reparsed. You should use DocumentBuilder#build instead. Note that the created documents should always be part of the LangiumDocuments service. You can always add them via addDocument.

[ballcoach12]

I was able to pull off what you suggest, @msujew. I don't have to make a call to update() on documentBuilder at all. Here is the final solution for reference:

export function addUnresolvedImportCollectionPhase(services: BlueframeDataTransformationServicesServices): void {
    const documentBuilder = services.shared.workspace.DocumentBuilder;
    documentBuilder.onBuildPhase(DocumentState.IndexedContent, async (docs) => {
        // findUnresolvedImports returns only URIs for packages we don't already have
        // It is a TODO to determine when a deployed package changes and how we invalidate and reload it
        const imports = findUnresolvedImports(docs, services);
        for (const imp0rt of imports) {
            const document = await services.references.ImportScopeResolver.getDocument(imp0rt);
            services.shared.workspace.LangiumDocuments.addDocument(document);
        }        
    })
}

[msujew]

Ha, interesting. I'm a bit confused, since these documents shouldn't get indexed. But if it works, it works ¯\_(ツ)_/¯

[ballcoach12]

I'll double-check that. I've checked my global scope computation, though, and the elements are there for sure, and references to elements in the imported documents get resolved. My ImportScopeResolver.getDocument() method looks like this:

async getDocument(uri: URI): Promise<LangiumDocument<DtsPackageRoot>> {
        // a real provider will make a call to the function registry to obtain the document
        // at the given URI.  Here, we just fake in a package
        const documentBuilder = this.services.shared.workspace.DocumentBuilder;           
        const document = this.services.shared.workspace.LangiumDocumentFactory.fromString<DtsPackageRoot>(sally, uri);
        await documentBuilder.build([document], {validation: true});
        return document;
    }

The provider shown here fakes in some code to import, but this gives you an idea of how I'm handling the docs.

[msujew]

Oh I see. Calling documentBuilder.build will of course build those documents. Yeah, that makes sense then. I thought you didn't call the build method at all. 👍

[ballcoach12]

@msujew, a couple of questions. Sorry for my lack of expertise here.

1. What would be the criteria for the findUnresolvedImports() function? In other words, what constitutes an unresolved import in this case?
2. What is the purpose of the newDocuments variable? I don't see it used here, and it isn't clear to me what its type would be. I assume that it's a LangiumDocument.

Also, my general design on this feature is to follow the methodology described in the docs here. I assume that this is the approach that would be recommended for my case, with the exception of how I retrieve the resource for reference from my remote endpoint.

[msujew]

What would be the criteria for the findUnresolvedImports() function? In other words, what constitutes an unresolved import in this case?

An unresolved import points to a document (or multiple documents) that hasn't been loaded yet.

What is the purpose of the newDocuments variable? I don't see it used here, and it isn't clear to me what its type would be. I assume that it's a LangiumDocument.

I've updated the code snippet above. There was a small error. It was supposed to be used for the documentBuilder.update call.

[georg-schwarz]

Hey, we're doing something similar: loading documents that were not in the originally defined workspace.
We have followed this thread so far, and we learned the following:

1. Where to register the documentBuilder.onBuildPhase hook in the VSCode extension?

   We call the hook registration function in the createServices method to ensure its usage.

2. Do we need to lock the building and/or adding of the new documents? Or is it only necessary with the update method you used in your example above?

   I think we need a mutex, otherwise we add documents twice. We made the lock only around the document adding.

Feedback appreciated! Hope it helps others that go down that path as well! Below a code sample:

export function createJayveeServices(context: DefaultSharedModuleContext): {
  shared: LangiumSharedServices;
  Jayvee: JayveeServices;
} {
  const shared = inject(
    createDefaultSharedModule(context),
    JayveeGeneratedSharedModule,
    JayveeSharedModule,
  );
  const Jayvee = inject(
    createDefaultModule({ shared }),
    JayveeGeneratedModule,
    JayveeModule,
  );
  shared.ServiceRegistry.register(Jayvee);

  addDynamicFileImport(Jayvee); // <<< adds the hook

  return { shared, Jayvee };
}


export function addDynamicFileImport(services: JayveeServices): void {
  const documentBuilder = services.shared.workspace.DocumentBuilder;
  const importResolver = services.ImportResolver;

  documentBuilder.onBuildPhase(DocumentState.IndexedContent, async (docs) => {
    for (const doc of docs) {
      const model = doc.parseResult.value;
      if (!isJayveeModel(model)) {
        return;
      }
      const importURIs = importResolver.findUnresolvedImportURIs(model);

      for (const importURI of importURIs) {
        await loadDocumentFromFs(importURI, services);
      }
    }
  });
}

async function loadDocumentFromFs(
  importURI: URI,
  services: JayveeServices,
): Promise<void> {
  const allowedFileExtensions = services.shared.ServiceRegistry.all.flatMap(
    (e) => e.LanguageMetaData.fileExtensions,
  );
  const hasAllowedFileExtension = allowedFileExtensions.some((ext) =>
    importURI.fsPath.endsWith(ext),
  );
  if (!hasAllowedFileExtension) {
    return;
  }

  const langiumDocuments = services.shared.workspace.LangiumDocuments;
  const documentBuilder = services.shared.workspace.DocumentBuilder;
  const documentFactory = services.shared.workspace.LangiumDocumentFactory;

  const file = await loadFileFromUri(importURI, services);
  if (file === undefined) {
    return;
  }

  const document = documentFactory.fromString<JayveeModel>(file, importURI);
  await documentBuilder.build([document], { validation: true });

  await services.shared.workspace.WorkspaceLock.write(() => {
    if (!langiumDocuments.hasDocument(document.uri)) {
      langiumDocuments.addDocument(document);
    }
  });
}

async function loadFileFromUri(
  uri: URI,
  services: JayveeServices,
): Promise<string | undefined> {
  const fileSystemProvider = services.shared.workspace.FileSystemProvider;

  try {
    return await fileSystemProvider.readFile(uri);
  } catch (e) {
    return undefined;
  }
}

[ballcoach12]

@georg-schwarz sorry for the late reply. I just saw your question.

In our implementation, we create and register a module for import scopes. It looks like this:

export interface ImportScopeResolver {

    /**
     * Retrieves the package from the package source
     * @param fqn the fully-qualified name of the package to retrieve
     * @returns a Promise for a {@link LangiumDocument} whose root node is a {@link DtsPackage};
     * or `undefined` if the package cannot be retrieved
     */
    getPackage(fqn: FullyQualifiedName): Promise<LangiumDocument<DtsPackageRoot> | undefined>
}



/**
 * Adds a hook into the document build phase that forces the retrieval and parsing of imported DTS packages
 * and places them in the index manager so that the package elements can be scoped.
 * 
 * @param services a BlueframeDataTransformationServicesServices instance that defines an {@link ImportScopeResolver}
 * that can retrieve imported packages
 */
export function addUnresolvedImportCollectionPhase(services: BlueframeDataTransformationServicesServices): void {
    const documentBuilder = services.shared.workspace.DocumentBuilder;
    documentBuilder.onBuildPhase(DocumentState.IndexedContent, async (docs) => {
        // findUnresolvedImports returns only URIs for packages we don't already have
        // It is a TODO to determine when a deployed package changes and how we invalidate and reload it
        const imports = findUnresolvedImports(docs, services);
        for (const imp0rt of imports) {
            const document = await services.references.ImportScopeResolver.getPackage(parseUri(imp0rt.uri));
            if (document) {
                imp0rt.import.resolves = true;
                if (!services.shared.workspace.LangiumDocuments.hasDocument(document.uri)) {
                    services.shared.workspace.LangiumDocuments.addDocument(document);
                }
            }
            else {
                console.warn(`document could not be obtained for package ${imp0rt.toString()}`)
            }
        }
    })
}

function buildFqn(i: ImportedPackage): FullyQualifiedName {
    let version: string | Version = LATEST

    if (i.version !== undefined) {
        const parts = i.version?.split(',');
        if (parts.length === 3) {
            const maj = parseInt(parts[0]);
            const min = parseInt(parts[1]);
            const rev = parseInt(parts[2]);

            if (isNaN(maj) || isNaN(min) || isNaN(rev)) {
                console.log(`Could not parse version into FQN: invalid version for package ${i.package}: ${i.version}.`);
            }
            else {
                version = {
                    "majorVersion": maj,
                    "minorVersion": min,
                    "revision": rev
                };
            }
        }
    }

    return {
        packageName: i.package,
        version: version,
        extension: '.dts'
    }
}

/**
* Processes a list of {@link LangiumDocument}s to provide a list of imports to resolve
* @param docs a list of {@link LangiumDocument}s that contain imports to resolve
* @returns an array of {@link Import} definitions for packages that should be imported
* into the global scope
*/
function findUnresolvedImports(docs: LangiumDocument<AstNode>[], services: LangiumServices): Array<ImportDescription> {
    const imports: ImportDescription[] = [];
    for (const doc of docs) {
        const root = doc.parseResult.value as DtsPackageRoot;
        if (root.package) {
            const rootNode = root.package;
            for (const i of rootNode.imports) {
                const fqn = buildFqn(i)
                const _uri = buildMemoryURI(services, fqn);

                if (services) {
                    const buildinURI = `file:///${fqn.packageName}-${fqn.version}${services.LanguageMetaData.fileExtensions[0]}`;
                    if (services.shared.workspace.LangiumDocuments.hasDocument(URI.parse(buildinURI))) {
                        i.resolves = true;
                        continue;
                    }
                    if (!(services.shared.workspace.LangiumDocuments.hasDocument(_uri))) {
                        imports.push({
                            uri: _uri,
                            import: i
                        });
                    }
                }
            }
        }
    }
    return imports;
}

interface ImportDescription {
    uri: URI,
    import: ImportedPackage
}

then we calladdUnresolvedImportCollectionPhase in the language module:

/**
 * Create the full set of services required by Langium.
 *
 * First inject the shared services by merging two modules:
 *  - Langium default shared services
 *  - Services generated by langium-cli
 *
 * Then inject the language-specific services by merging three modules:
 *  - Langium default language-specific services
 *  - Services generated by langium-cli
 *  - Services specified in this file
 *
 * @param context Optional module context with the LSP connection
 * @returns An object wrapping the shared services and the language-specific services
 */
export function createDtsAndJsonQueryServices(context: DefaultSharedModuleContext): {
    shared: LangiumSharedServices,
    dtsServices: BlueframeDataTransformationServicesServices
} {

    const shared = inject(
        createDefaultSharedModule(context),
        DtsAndJsonQueryGeneratedSharedModule,
        XetaSharedModule
    );
    const dtsServices = inject(
        createDefaultModule({ shared }),
        BlueframeDTSGeneratedModule,
        BlueframeDataTransformationServicesModule
    );
    addValidationNotificationPhase(dtsServices);
    addUnresolvedImportCollectionPhase(dtsServices);
    if (context.connection) {
        addExecuteCommandHandler(context.connection, shared)
    }
    shared.ServiceRegistry.register(dtsServices);
    return { shared, dtsServices };
}

Finally, we have an import-scope-resolver that make the call to our external resource to fulfill scopes:

export class DtsImportScopeResolver implements ImportScopeResolver {
  services: BlueframeDataTransformationServicesServices;

  constructor(services: BlueframeDataTransformationServicesServices) {
    this.services = services;
  }

  async getPackage(fqn: FullyQualifiedName): Promise<LangiumDocument<DtsPackageRoot> | undefined> {
    try {
      const data = {
        majorVersion: (fqn.version as Version).majorVersion,
        minorVersion: (fqn.version as Version).minorVersion,
        packageName: fqn.packageName,
        revision: (fqn.version as Version).revision
      }
      getPackageFqn(data).then(async (modelsPackage) => {
        if (modelsPackage.instances && modelsPackage.instances[0].code) {
          // parse the response
          const documentBuilder = this.services.shared.workspace.DocumentBuilder;
          const document =
            this.services.shared.workspace.LangiumDocumentFactory.fromString<DtsPackageRoot>(
              decodePackageCode(modelsPackage.instances[0].code),
              buildMemoryURI(this.services, fqn)
            );
          await documentBuilder.build([document], { validation: true });
          return document;
        } else {
          return undefined;
        }
      });


    }
    catch (e) {
      console.error(`scoping: Received non-200 response from package import call: ${e}`);
      return undefined;
    }
  }
}

getPackage() is called in the addUnresolvedImportCollectionPhase() function.

Let me know if you have further questions.