Kafka Connect BigQuery Connector

This is an implementation of a sink connector from Apache Kafka to Google BigQuery, built on top of Apache Kafka Connect.

History

This connector was originally developed by WePay. In late 2020 the project moved to Confluent, with both companies taking on maintenance duties. In 2024, Aiven created its own fork based off the Confluent project in order to continue maintaining an open source, Apache 2-licensed version of the connector.

Configuration

Sample

An example connector configuration, that reads records from Kafka with JSON-encoded values and writes their values to BigQuery:

{
  "connector.class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
  "topics": "users, clicks, payments",
  "tasks.max": "3",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",

  "project": "kafka-ingest-testing",
  "defaultDataset": "kcbq-example",
  "keyfile": "/tmp/bigquery-credentials.json"
}

Complete docs

See here for a list of the connector's configuration properties.

Download

Releases are available in the GitHub release tab.