Versatile Data Kit 0.13

[antoniivanov]

Major features include:

New plugin: vdk-gdp-execution-id

An installed Generative Data Pack plugin automatically expands the data sent for ingestion.

This GDP plugin detects the execution ID of a Data Job running, and decorates your data product with it. So that,
it is now possible to correlate a data record with a particular ingestion Data Job execution ID.

For more information see the plugin documentation

vdk-dag: pass arguments to jobs in a DAG

Now each job in a DAG can be passed arguments :

{
"job_name": "name-of-job",
"team_name": "team-of-job",
"fail_meta_job_on_error": false,
"arguments": <ARGUMENTS IN DICTIONARY FORMAT HERE>,
"depends_on": ["name-of-job1", "name-of-job2"]
}

vdk-notebook: VDK job input in vdk cells

Users will be able to develop jobs entirely in a Notebook file with all features of VDK available out of the box
After installation of vdk-notebook users can now will have access to job_input interface to execute templates, ingest data and all else.

vdk-notebook: vdk and non-vdk cells

To enable separation of product and development code vdk-notebook integration provides a way for users to set which cells are deployable and part of their production code and which are not.

quickstart-vdk now includes the Operations UI

When installing quickstart-vdk VDK Server is available for local testing and now includes UI:

pip install quickstart-vdk
vdk server --install 

For more information see here

Versatile Data Kit Frontend npm libraries release

The Versatile Data Kit Frontend provides 2 npm (angular) libraries which can be used to build integrate VDK UI with your own screens:

• @versatiledatakit/data-pipelines
  Versatile Data Kit Data Pipelines library provides UI screens that helps to manage data jobs via Versatile Data Kit Control Service
• @versatiledatakit/shared
  Versatile Data Kit Shared library enables reusability of shared features like: NgRx Redux, Error Handlers, Utils, Generic Components, etc.

What's Changed

• control service: Add supported python version configuration by @doks5 in Control Service: Add supported python version configuration #1761
• control-service: fix python api release by @murphp15 in Control-service: fix python api release #1946
• control service: Dynamically set job base image in builder by @doks5 in control service: Dynamically set job base image in builder #1864
• control-service: Add python_version to Control Service API by @doks5 in control-service: Add python_version to Control Service API #1806
• control-service: Add python_version to Execution API by @mivanov1988 in control-service: Add python_version to Execution API #1878
• control-service: Add python_version to GraphQL API by @mivanov1988 in control-service: Add python_version to GraphQL API #1909
• control-service: Add support for Python 3.11 by @mivanov1988 in control-service: Add support for Python 3.11 #1861
• control-service: Dynamically set vdk image in JobImageDeployer by @doks5 in control-service: Dynamically set vdk image in JobImageDeployer #1883
• control-service: Expose supported python versions in helm by @doks5 in control-service: Expose supported python versions in helm #1935
• control-service: Remove support for very old k8s apiVersion by @murphp15 in control-service: Remove support for very old k8s apiVersion #1860
• control-service: add the frontend to helm by @murphp15 in control-service: add the frontend to helm #1885
• control-service: enable usage of aws temporary credentials by @mrMoZ1 in control-service: enable usage of aws temporary credentials #1787
• control-service: expose supported python versions by @dakodakov in control-service: expose supported python versions #1841
• control-service: fix failing image publisher by @murphp15 in control-service: fix failing image publisher #1810
• control-service: force job builder deploy by @mrMoZ1 in control-service: force job builder deploy #1823
• control-service: new helm release by @murphp15 in control-service: new helm release #1910
• control-service: revert job builder python version by @mrMoZ1 in control-service: revert job builder python version #1840
• control-service: update helm charts for service account credentials by @mrMoZ1 in control-service: update helm charts for service account credentials #1800
• control-service: update job builders for aws temporary credentials by @mrMoZ1 in control-service: update job builders for aws temporary credentials #1799
• documentation: VDK components explained by @ivakoleva in documentation: VDK components explained #1865
• frontend: Align code formatting in frontend projects by @gorankokin in frontend: Align code formatting in frontend projects #1863
• frontend: Configurable OAuth by @ivakoleva in frontend: Configurable OAuth #1913
• frontend: Update docs with build/test configuration by @DeltaMichael in frontend: Update docs with build/test configuration #1928
• frontend: add build.sh by @tozka in frontend: add build.sh #1807
• frontend: fix npm lint warnings by @DeltaMichael in frontend: fix npm lint warnings #1808
• frontend: increase the amount of resources for build in cicd by @murphp15 in frontend: increase the amount of resources for build in cicd #1931
• frontend: prepare for official release shared and dp libs by @gorankokin in frontend: prepare for official release shared and dp libs #1795
• frontend: publish docker image for ui by @DeltaMichael in frontend: publish docker image for ui #1872
• frontend: remove unused config in helm chart for frontend dns by @murphp15 in frontend: remove unused config in helm chart for frontend dns #1932
• frontend: Stabilization for e2e tests by @gorankokin in frontend: Stabilization for e2e tests #1876
• frontend: Auth configurations organized by @ivakoleva in frontend: Auth configurations organized #1957
• frontend: change history link in data job by @gorankokin in frontend: change history link in data job #1884
• specs: VEP-1739 Update status and reorganise document by @doks5 in specs: VEP-1739 Update status and reorganise document #1857
• specs: VEP-1739 updated API section by @mivanov1988 in specs: VEP-1739 updated API section #1882
• specs: update Multiple Python versions VEP summary by @tozka in specs: update Multiple Python versions VEP summary #1792
• vdk-vep: update vep status by @dakodakov in vdk-VEP-1493: update vep status #1951
• vdk-cicd: apply limit ranges for storage by @tozka in vdk-cicd: apply limit ranges for storage #1815
• vdk-cicd: set ephemeral storage request/limits by @tozka in vdk-cicd: set ephemeral storage request/limits #1813
• vdk-control-cli: fix circular import dependecy by @tozka in vdk-control-cli: fix circular import dependecy #1820
• vdk-control-cli: refactor output printing with printer class by @tozka in vdk-control-cli: refactor output printing with printer class #1819
• vdk-control-cli: use assert_click_status by @tozka in vdk-control-cli: use assert_click_status #1817
• vdk-control-cli: use common output printer by @tozka in vdk-control-cli: use common output printer #1852
• vdk-control-cli: vdk list -mmm to return executions by @tozka in vdk-control-cli: vdk list -mmm to return executions #1818
• vdk-control-service: publish python client library by @dakodakov in vdk-control-service: publish python client library #1934
• vdk-dags: improve DAGs user-facing documentation by @yonitoo in vdk-dags: improve DAGs user-facing documentation #1892
• vdk-gdp-execution-id: a Generative Data Pack expanding with execution ID by @ivakoleva in vdk-gdp-execution-id: a Generative Data Pack expanding with execution ID #1877
• vdk-gdp-execution-id: import fix by @ivakoleva in vdk-gdp-execution-id: import fix #1961
• vdk-github-workflows: ubuntu latest update by @ivakoleva in vdk-github-workflows: ubuntu latest update #1943
• vdk-jupyter: UI test enhancements by @duyguHsnHsn in vdk-jupyter: UI test enhancements #1783
• vdk-jupyter: add UI vdk cell marks by @duyguHsnHsn in vdk-jupyter: add UI vdk cell marks #1891
• vdk-jupyter: job run messages by @duyguHsnHsn in vdk-jupyter: job run messages #1908
• vdk-jupyter: remove react-test-renderer package from package.json by @duyguHsnHsn in vdk-jupyter: remove react-test-renderer package from package.json #1881
• vdk-lineage: support for latest version sqllineage library by @tozka in vdk-lineage: support for latest version sqllineage library #1816
• vdk-meta-jobs: Meta Jobs DAG validation by @yonitoo in vdk-meta-jobs: Meta Jobs DAG validation #1785
• vdk-meta-jobs: add DAG with args example by @yonitoo in vdk-meta-jobs: add DAG with args example #1859
• vdk-meta-jobs: add some configurable variable references in the VEP by @yonitoo in vdk-meta-jobs: add some configurable variable references in the VEP #1794
• vdk-meta-jobs: exec job with arguments by @yonitoo in vdk-meta-jobs: exec job with arguments #1839
• vdk-meta-jobs: fix DAG image in example by @yonitoo in vdk-meta-jobs: fix DAG image in example #1920
• vdk-meta-jobs: improve DAGs code documentation by @yonitoo in vdk-meta-jobs: improve DAGs code documentation  #1873
• vdk-metajobs: Deprecate plugin by @gageorgiev in vdk-metajobs: Deprecate plugin #1930
• vdk-notebook: add hook for saving error information into json file by @duyguHsnHsn in vdk-notebook: add hook for saving error information into json file #1842
• vdk-plugins: add connection hook activity diagram by @tozka in vdk-plugins: add connection hook activity diagram #1786
• vdk-plugins: test only oldest and newest supported python version by @tozka in vdk-plugins: test only oldest and newest supported python version #1811
• vdk-server: quickstart vdk now includes the UI by @murphp15 in vdk-server: quickstart vdk now includes the UI #1912

New Contributors

• @gorankokin made their first contribution in frontend: prepare for official release shared and dp libs #1795

Full Changelog: v0.12...v0.13

---

This discussion was created from the release Versatile Data Kit 0.13.