Versatile Data Kit 0.6

[antoniivanov]

Summary

Major features include:

Configuration auto-wiring improvement: detect non vdk_ prefixed environment variables

Before configuration option must have been prefixed with "vdk_" when set as an environment variable in order to be recognized.
This was very error prone since the options are documented without the prefix.

Now they can be set without a prefix as well.

The following are equivalent:

export VDK_DB_DEFAULT_TYPE='impala'
export DB_DEFAULT_TYPE='impala'

If both are set, the "prefixed" variable has a higher priority.

New plugin/library: vdk-lineage-model

VDK Lineage Model plugin aims to abstract emitting lineage data from VDK data jobs, so that different lineage loggers can be configured at run time in any plugin that supports emitting lineage data

Check out more at the plugin page.

New export-csv command

Alongside vdk ingest-csv which enabled users to import (or ingest) CSV data into a table.
Users can now export CSV with a simple command from SQL query:

vdk export-csv -q "select * from my_table --file 'output.csv'

Checkout out more at the plugin page

In memory properties client

Until now properties required Control Service to be able to work. Sometimes for prototyping and testing purposes, you do not need to connect to external services.

• New configuration value can be set.

In a specific job's config file (config.ini

[vdk]
properties_default_type = memory

Or as an environment variable

export properties_default_type="memory"

• Now the properties would be entirely in memory. That means they will be "deleted" after the job's run.

New example: Ingest and anonymize

Example how to anonymize any data being ingested using VDK with a plugin.

Check out more at the example page

New example: Airflow integration

Example how to create dependencies between data job in Airflow.

Check out more at the example page

Package versions

See installation instructions here.
The versions of VDK components released under VDK 0.6 are:

Main components

control-service 1.5.620438292
vdk-core==0.3.620677184

Plugins

airflow-provider-vdk==0.0.602273476
vdk-lineage-model== 0.0.581430542
vdk-kerberos-auth==0.3.584577337
vdk-ingest-http==0.2.616713987
vdk-impala==0.4.613570906
vdk-lineage== 0.3.604201902
vdk-trino== 0.4.605101952

What's Changed

• airflow-provider-vdk: Add hidden fields to VDK Connection by @doks5 in [vdk-plugins] airflow-provider-vdk: Add hidden fields to VDK Connection #883
• control-service: Atomic job cancellation by @gageorgiev in control-service: Atomic job cancellation #860
• control-service: Fluentd integration for data jobs by @mivanov1988 in Fluentd integration for data jobs #940
• control-service: Secure job builder image by @gageorgiev in control-service: Secure job builder image #936
• control-service: add default jwt jwk uri by @mrMoZ1 in control-service: add default jwt jwk uri #873
• control-service: fix the examples in swagger by @tozka in control-service: fix the examples in swagger #945
• control-service: fix vdk-server startup issues by @mrMoZ1 in control-service: fix vdk-server startup issues #908
• control-service: increase integration test builder memory by @mrMoZ1 in control-service: increase integration test builder memory #929
• control-service: upgrade docker container used in cicd by @mrMoZ1 in control-service: upgrade docker container used in cicd #911
• vdk-airflow: populate readme by @tozka in vdk-airflow: populate readme #924
• vdk-control-cli: remove hidden flag for CLI commands by @tozka in vdk-control-cli: remove hidden flag for CLI commands #902
• vdk-control-cli: use latest dependencies version during build by @tozka in vdk-control-cli: use latest dependencies version during build #903
• vdk-core,vdk-impala,vdk-lineage,vdk-trino: Support for pluggy 1.0 by @gageorgiev in vdk-core,vdk-impala,vdk-lineage,vdk-trino: Support for pluggy 1.0 #931
• vdk-core: Add printed output to set-default and reset-default by @gageorgiev in vdk-core: Add printed output to set-default and reset-default #884
• vdk-core: BaseVdkError exception propagation flaw fix by @ivakoleva in vdk-core: BaseVdkError exception propagation flaw fix #917
• vdk-core: Improve ingestion error logging by @gageorgiev in vdk-core: Improve ingestion error logging #930
• vdk-core: add memory properties client by @tozka in vdk-core: add memory properties client #921
• vdk-core: add option to disable version check by @tozka in vdk-core: add option to disable version check #876
• vdk-core: detect non vdk_ prefixed environment values for config by @tozka in vdk-core: detect non vdk_ prefixed environment values for config #874
• vdk-core: execution result missing exception and blamee fix by @ivakoleva in vdk-core: execution result missing exception and blamee fix #938
• vdk-core: hide native cursor from execute hook by @tozka in vdk-core: hide native cursor from execute hook #875
• vdk-core: make db_default_type case insensitive by @tozka in vdk-core: make db_default_type case insensitive #935
• vdk-core: show log_level_vdk in help by @tozka in vdk-core: show log_level_vdk in help #905
• vdk-core: step loading failure misclassified as Platform error fix by @ivakoleva in vdk-core: step loading failure misclassified as Platform error fix #920
• vdk-core: termination message now idempotent by @mrMoZ1 in vdk-core: termination message now idempotent #909
• vdk-core: vdk_exception hook exit code fix by @ivakoleva in vdk-core: vdk_exception hook exit code fix #912
• vdk-core: vdk_exception hook exit code fix by @ivakoleva in vdk-core: vdk_exception hook exit code fix #915
• vdk-csv: add export-csv command by @duyguHsnHsn in vdk-csv: add export-csv command #934
• vdk-examples: add ingest and anonymize example by @tozka in vdk-examples: add ingest and anonymize example #922
• vdk-impala, vdk-trino: Remove deprecated use of result field by @gageorgiev in vdk-impala, vdk-trino: Remove deprecated use of result field #933
• vdk-impala: Add performance logs by @VladimirPetkov1 in vdk-impala: Add performance logs #939
• vdk-impala: Add support for lineage in vdk-impala by @VladimirPetkov1 in vdk-impala: Add support for lineage in vdk-impala #932
• vdk-ingest-http: reduce verbosity of ingestion logs by @tozka in vdk-ingest-http: reduce verbosity of ingestion logs #943
• vdk-kerberos-auth: Separate async event loop by @doks5 in [vdk-plugins] vdk-kerberos-auth: Separate async event loop #885
• vdk-lineage-model: Extract Lineage Model in separate plugin by @VladimirPetkov1 in vdk-lineage-model: Extract Lineage Model in separate plugin #896
• vdk-server: Pin kubernetes API version by @doks5 in [vdk-plugins] vdk-server: Pin kubernetes API version #919
• vdk-server: fix for vdk server crashing on startup by @mrMoZ1 in vdk-server: fix for vdk server crashing on startup #907
• vdk-trino, vdk-linage: Switch to vdk-lineage-model by @VladimirPetkov1 in vdk-trino, vdk-linage: Switch to vdk-lineage-model #898
• vdk-trino: fix broken tests by @tozka in vdk-trino: fix broken tests #900
• versatile-data-kit: Add Data lifecycle image and minor changes by @zverulacis in Add Data lifecycle image and minor changes #887
• versatile-data-kit: Add getting started, ask for help, PR checklist by @zverulacis in Add getting started, ask for help, PR checklist #881
• versatile-data-kit: Add intro part to contributing.md from the template by @zverulacis in Add intro part to contributing.md from the template #880
• versatile-data-kit: Airflow Documentation by @gageorgiev in versatile-data-kit: Airflow Documentation #857
• versatile-data-kit: add link to csv example doc by @tozka in versatile-data-kit: add link to csv example doc #893
• versatile-data-kit: add logo image by @tozka in versatile-data-kit: add logo image #877
• versatile-data-kit: make easier slack instructions by @tozka in versatile-data-kit: make easier slack instructions #925
• versatile-data-kit: update link in examples by @tozka in versatile-data-kit: update link in examples #892
• versatile-data-kit: update logo for dark mode by @tozka in versatile-data-kit: update logo for dark mode #878

New Contributors

• @VladimirPetkov1 made their first contribution in vdk-lineage-model: Extract Lineage Model in separate plugin #896
• @duyguHsnHsn made their first contribution in vdk-csv: add export-csv command #934

Full Changelog: v0.5...v0.6

---

This discussion was created from the release Versatile Data Kit 0.6.