Releases: GoogleCloudPlatform/DataflowTemplates
Dataflow Templates 2022-04-25-00_RC00
Release Week of 2022-04-25
New Templates
[SpannerChangeStreamsToGcs] Implement Spanner Change Streams to BigQuery changelog table replication Dataflow template
Improvements
[PubSubToSplunk] Add parameter to allow users to specify if HTTP messages sent to HEC endpoint should be GZIP encoded.
Bug Fixes
[DatastreamToSQL] Use KV into the write
Contributors
Contributors
Assets 2
Dataflow Templates 2022-04-18-00_RC00
Release Week of 2022-04-18
Improvements
- [JDBC to BigQuery] Add documentation for
disabledAlgorithmsoption
Bug Fixes
- [Datastream to SQL] Prevent Oracle records from being marked stale when multiple records with the same timestamp and SCN are loaded
Contributors
Contributors
Assets 2
Dataflow Templates 2022-04-11-00_RC00
Release Week of 2022-04-11
New Templates
- Spanner Change Streams to GCS (only available in repository)
Improvements
- [Datastream to BigQuery] Use stateful transform to handle Oracle rollbacks
- [Datastream to BigQuery] Use txn id and row id for unique rollback values
- [Dataplex JDBC Ingestion] Added
writeDispositionoption to allow setting a write strategy if file already exists - [Spanner Import/Export] Support building change streams DDL from information schema
Bug Fixes
- [Datastream to BigQuery] Change DATETIME handling to account for local timezones in Oracle in replication
- [Dataplex Tiering] Fix TIME field handling
Contributors
- dhercher
- nancyxu123
- olegsa
- pranavbhandari
- 1 internal contributor
Dataflow Templates 2022-04-04-00_RC00
Release Week of 2022-04-04
Improvements
[DataStreamToSQL] Improved logging
[DatastreamToBigQuery] Added datastream logging for json failures
Bug Fixes
[All JDBC Templates] Fix bug that handles JDBC CLOB data converting to String
[Jdbc To BigQuery] Fix bug to get specific Java object from JDBC data type Date and timestamp
Contributors
dhercher
weiwenxu
zhoufek
Dataflow Templates 2022-03-28-00_RC00
Release Week of 2022-03-28
Improvements
Upgraded to Beam 2.37.
[DataStreamToSQL] Added failing SQL statement to error message logged.
[DataStreamToBigQuery] Added an option to add pane info to a windowed filename.
Added first integration test StreamingDataGeneratorIT.
Bug Fixes
Fixed flaky testAssetWithEntityJsonToParquetFailOnExistingFilesE2E test.
Contributors
dhercher
olegsa
pranavbhandari
zhoufek
Dataflow Templates 2022-03-21-00_RC01
Release Week of 2022-03-21
New Templates
Deprecated Datastream to Postgres template (replaced with Datastream to SQL template)
Improvements
[Datastream to SQL] Added support for specifying log level and logging SQL in debug mode
[Spanner to Text] Cleaned up and removed LocalSpannerIO
Bug Fixes
[Pub/Sub to Splunk] Fixed write latency metrics
Contributors
dhercher
pranavbhandari
Dataflow Templates 2022-03-14-00_RC00
Release Week of March 14, 2022 🥧
New Templates
N/A
Improvements
[Pub/Sub to Splunk] Added secret manager support
Bug Fixes
[Datastream Oracle] Map DATE -> DATE instead of DATE -> TIMESTAMP
New Regions
N/A
Contributors
pranavbhandari
Dataflow Templates 2022-03-07-00_RC00
Release Week of March 7, 2022
New Templates
N/A
Improvements
[Pub/Sub to Splunk] Improved metrics
Bug Fixes
N/A
New Regions
N/A
Contributors
pranavbhandari
Dataflow Templates 2022-02-28-00_RC00
Release Week of February 28, 2022
New Templates
N/A
Improvements
[Datastream to SQL] Added support for custom connection strings. This can be used to enable SSL for example, though overriding the sslcert/sslrootcert/sslkey parameters (which expect a local file path) is not yet supported.
Bug Fixes
N/A
New Regions
N/A
Contributors
dhercher
zhoufek
Dataflow Templates 2022-02-21-00_RC00
Release Week of February 21, 2022
New Templates
N/A
Improvements
[All Templates] Upgrade beam version to 2.36
[All Dataplex Templates] Refactor common code
Bug Fixes
[Dataplex: BigQuery to GCS] Dataplex Tiering template: Refactor BigQueryTableToGcsTransform date type fix. This should not change the visible behavior of the transform.
New Regions
N/A
Contributors
olegsa
pranavbhandari
zhoufek