v4.44.0

For installation instructions check out the getting started guide.

Added

• Go API: New public/license package added to allow custom programmatic instantiations of Redpanda Connect to run enterprise license components. (@Jeffail)

Fixed

• gcp_bigquery output with parquet format no longer returns errors incorrectly. (@rockwotj)
• postgres_cdc input now allows quoted identifiers for the table names. (@mihaitodor, @rockwotj)

The full change log can be found here.

v4.43.1

For installation instructions check out the getting started guide.

Fixed

• Trial Redpanda Enterprise licenses are now considered valid. (@Jeffail)
• The redpanda_migrator_bundle output now skips schema ID translation when translate_schema_ids: false and schema_registry is configured. (@mihaitodor)

The full change log can be found here.

v4.43.0

For installation instructions check out the getting started guide.

Changed

• The pg_stream input has been renamed to postgres_cdc. The old name will continue to function as an alias. (@rockwotj)
• The postgres_cdc input no longer emits mode metadata and instead snapshot reads set operation metadata to be read instead of insert. (@rockwotj)

Fixed

• The redpanda_migrator_bundle output no longer attempts to translate schema IDs when a schema registry is not configured. (@mihaitodor)

The full change log can be found here.

v4.42.0

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@le-vlad)
• Add support for Parquet files to bigquery output (@rockwotj)
• (Benthos) New exists operator added to the cache processor. (@mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@Jeffail)

The full change log can be found here.

v4.42.0-rc1

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@le-vlad)
• Add support for Parquet files to bigquery output (@rockwotj)
• (Benthos) New exists operator added to the cache processor. (@mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@Jeffail)

The full change log can be found here.

v4.41.0

For installation instructions check out the getting started guide.

Added

• Field max_records_per_request added to the aws_sqs output. (@Jeffail)

Fixed

• (Benthos) Fixed an issue where running a CLI with a custom environment would cause imported templates to be rejected. (@Jeffail)

Changed

• The -cgo suffixed docker images are no longer built and pushed along with the regular images. This decision was made due to low demand, and the unacceptable cadence with which the image base (Debian) receives security updates. It is still possible to create your own CGO builds with the command CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. (@Jeffail)

The full change log can be found here.

v4.40.0

For installation instructions check out the getting started guide.

Added

• New pg_stream input supporting change data capture (CDC) from PostgreSQL. (@le-vlad)
• Field metadata_max_age added to the redpanda_migrator_offsets output. (@mihaitodor)
• Field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator outputs. (@mihaitodor)
• (Benthos) New Bloblang method timestamp. (@mihaitodor)
• (Benthos) New benchmark processor. (@ooesili)

Fixed

• Addresses an issue where snowflake_streaming could create more channels than configured. (@rockwotj)

Changed

• The snowflake_streaming output with schema_evolution.enabled set to true can now autocreate tables. (@rockwotj)
• Fields translate_schema_ids and schema_registry_output_resource added to the redpanda_migrator output. (@mihaitodor)
• Fields backfill_dependencies and input_resource added to the schema_registry output. (@mihaitodor)
• The schema_registry input and output and the schema_registry_encode and schema_registry_decode processors now use the github.com/twmb/franz-go/pkg/sr SchemaRegistry client. (@mihaitodor)
• Metadata field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs now contains a unix timestamp with millisecond precision. (@mihaitodor)
• Metadata field kafka_timestamp removed from the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs. (@mihaitodor)

The full change log can be found here.

v4.39.0

For installation instructions check out the getting started guide.

Added

• New timeplus input. (@ye11ow)
• New snowflake_streaming output. (@rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@ooesili)
• New CLI flag --secrets added. (@Jeffail)
• New CLI flag --disable-telemetry added. (@Jeffail)
• New experimental spicedb watch input. (@simon0191)
• New redpanda_common input and output. (@Jeffail)
• New redpanda input and output. (@Jeffail)
• New snowflake_streaming output. (@rockwotj)

Fixed

• The kafka, kafka_franz and redpanda_migrator outputs no longer waste CPU for large batches. (@rockwotj)

Changed

• The aws_sqs output field url now supports interpolation functions. (@rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@Jeffail)

The full change log can be found here.

v4.39.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• New timeplus input. (@ye11ow)
• New snowflake_streaming output. (@rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@ooesili)
• New CLI flag --secrets added. (@Jeffail)
• New CLI flag --disable-telemetry added. (@Jeffail)

Changed

• The aws_sqs output field url now supports interpolation functions. (@rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@Jeffail)

The full change log can be found here.

v4.38.0

For installation instructions check out the getting started guide.

Added

• Anonymous telemetry data is now sent by Connect instances after running for >5 mins. Details about which data is sent, when it is sent, and how to disable it can be found in the telemetry README. (@Jeffail)
• Field checksum_algorithm added to the aws_s3 output. (@dom-lee-naimuri)
• Field nkey added to nats, nats_jetstream, nats_kv and nats_stream components. (@ye11ow)
• Field private_key added to the snowflake_put output. (@mihaitodor)
• New azure_data_lake_gen2 output. (@ooesili)
• New timeplus output. (@ye11ow)

Fixed

• The elasticsearch output now performs retries for HTTP status code 429 (Too Many Requests). (@kahoowkh)
• The docs for the collection field of the mongodb output now specify support for interpolation functions. (@mihaitodor)

Changed

• All components with a default path field value (such as the aws_s3 output) containing the deprecated function count have now been changed to use the new function counter. This could potentially change behaviour in cases where multiple components are executing a mapping with a count function sharing the same of the old default count, and these counters need to cascade. This is an extremely unlikely scenario, but for all users of these components it is recommended that your path is defined explicitly, and in a future major version we will be removing the defaults.

The full change log can be found here.