Making use of Webhooks with the GithubOrgEntityProvider

[minkimcello]

Context

The current GithubOrgEntityProvider fetches members and teams of an organization. There's only one function available for updating the database and it's read() - it fetches the users/teams and does a "full" mutation to the database.

There should be two ways of triggering an update - periodically and whenever there's a new change. In #10 I added a task scheduler to run read() once a day. And alongside the task scheduler, we also want to be able to trigger an update whenever a member/team is added or removed so that we don't need to potentially wait a day for the backstage instance to reflect the new updates.

Although GithubOrgEntityProvider only fetches data about members and teams, the Github App we're using with backstage is configured to deliver webhooks on not only members/teams events but also repository events.

What this means is if we configure our catalog builder like so:

// packagse/backend/src/plugins/catalog.ts
...
export default async function createPlugin(env) {
  ...
  const gitProvider = GithubOrgEntityProvider.fromConfig(env, options);
  builder.addEntityProvider(gitProvider);
  const { processingEngine, router } = await builder.build();
  await processingEngine.start();

  router.post("/github/webhook", async (req, res) => {
    await gitProvider.read();
    res.send("Success!");
  }

  return router;
}

The entity provider will do a full mutation for the members/teams several times a day even if the webhook event has nothing to do with members/teams.

Possible Solution

There could be a deltaRead() function that takes an argument of relevant entities:

// GithubOrgEntityProvider.ts

export class GithubOrgEntityProvider implements EntityProvider {
  ...
  async connect() {...};
  async read() {...};
  
  async deltaRead(newEntity) {
    if (newEntity) {
      await this.connection.applyMutation({
        type: "delta",
        entity: newEntity
      })
    }
  }
}

That way, in our catalog builder, we can invoke an update to the database only when we need it to:

// packagse/backend/src/plugins/catalog.ts
export default async function createPlugin(env) {
  ...
  router.post("/github/webhook", async (req, res) => {
    if ( req.event == "member" || req.event == "team" ) {
      await gitProvider.deltaRead(req.payload);
      res.send("Success!");
    }
  }
  ...
}