Bilbo, the buildkite failed builds notifier

A Buildkite webhook that sends out an email to people that potentially have failed a build.

It looks like this:

Failures stack:

until the build passes, at which the culprits are reset.

It uses the Buildkite webhooks and webtask.io.

Assumptions

• The culprits are stored per pipeline, so only one branch can be used for each pipeline, e.g. master. If you want to track culprits between builds in multiple branches, you need to set up separate webtasks.
• Only the current person triggering the Buildkite pipeline will be notified, assuming all others have been notified in the builds before. This will not work for commits which use [skip ci] or [ci skip].
• Pipelines are tracked with their organization, so you can use one webtask with multiple organizations, even if you have pipelines with the same name.
• The commit shas, the commit message, the email and name of the triggering person and some other data is stored in the webtask storage. For what is stored, see here.
• TODO: in case an older build on the same pipeline that is passed after failure (by reruns for example), it removes the recorded failed state currently. Ideally the state would be kept, from the failing build upwards.

Installation

Via wt-cli:

wt create https://raw.githubusercontent.com/joscha/buildkite-failed-builds-notifier/master/failed-builds-notification.js \
          --name bilbo \
          --secret SENDGRID_API_TOKEN=<SENDGRID_API_TOKEN> \
          --secret MANDRILL_API_KEY=<MANDRILL_API_KEY> \
          --secret BUILDKITE_TOKEN=<BUILDKITE_TOKEN> \
          --secret SENDER_EMAIL_ADDRESS=me@domain.com \
          --dependency github-url-from-git@1.5.0 \
          --dependency sendgrid@5.2.3

You need either SENDGRID_API_TOKEN or MANDRILL_API_KEY. If both given, Mandrill will be preferred.

Needed modules

wt modules add github-url-from-git@1.5.0
wt modules add sendgrid@5.2.3
wt modules add mandrill-api@1.0.45

Needed secrets:

• Either SENDGRID_API_TOKEN: Create a new API token and give it full access for Mail Send:

• Or: MANDRILL_API_KEY: Create a new API key

• BUILDKITE_TOKEN: from your Buildkite webhook settings:

• SENDER_EMAIL_ADDRESS: The email address that is used by the email provider to generate the build fail emails.

Email settings

Sendgrid

Disable click tracking

otherwise the URLs in the plain text email become really long.

Enable plain content

otherwise the plain text will be automatically converted to HTML by sendgrid and the formatting will be off.

Buildkite webhook settings

Enable the ping and build.finished events:

Test

You can test the webtask with this snippet:

{
  "event": "build.finished",
  "build": {
    "web_url": "https://buildkite.com/some-org/some-pipeline/builds/1",
    "number": 1,
    "state": "failed",
    "message": "Some build message",
    "commit": "c0ffeec0ffeec0ffeec0ffeec0ffeec0ffeec0ff",
    "creator": {
      "name": "Mister Test",
      "email": "your@email.com"
    }
  },
  "pipeline": {
    "name": "Some pipeline",
    "url": "https://api.buildkite.com/v2/organizations/some-org/pipelines/some-pipeline",
    "slug": "some-pipeline",
    "repository": "git@github.com:some/repo.git"
  }
}

Important: Be sure to replace your@email.com with your own email, to actually receive anything.

Don't forget to send this as JSON (Content-Type: application/json) and also pass your Buildkite token via the X-Buildkite-Token header. You can replace "state": "failed" with "state": "passed" to simulate a passed build run.