Versatile Data Kit 0.12

[antoniivanov]

Major features include:

Open-sourcing VDK Operations UI

VDK Operations UI would enable data practitioners to efficiently manage (operate and monitor) their data jobs.
It has been used internally in VMware for some time and the team open source it last month.

Check out more details at the Operations UI VEP

Look forward to the official launch soon.

Documentation Improvements

Significantly simplified and improve the main README and the CONTRIBUTING.md thanks to @gary-tai and @zverulacis

VDK Meta Jobs Preparation for Alpha release

implemented a limit on starting jobs at once

META_JOBS_MAX_CONCURRENT_RUNNING_JOBS=<number>

Learn more about the VDK Meta Jobs features in VDK Meta Jobs VEP

Started initiative to support multiple python versions

We are working on introducing an optional python_version property to the Control Service API, which allows users to specify the Python version they want to use for their job deployment. This means users no longer have to rely on the service administrator to make changes to the configuration and can deploy their jobs with the version they need.

See more information in the Multiple Python Versions VEP

Started initiative to create Secrets Interface

So far the way VDK recommended to store secrets was to use Properties API. Though it works well, it doesn't really meet the criteria for storing properly restricted data and likely also confidential data

The team is working on providing similar to Properties interface Secrets (backed by HashiCorp Vault).

See more information in the Vault Integration For Secrets Storage VEP

What's Changed

• control-service: Update helm template license headers by @DeltaMichael in control-service: Update helm template license headers #1670
• control-service: don't include needless service token by @tozka in control-service: don't include needless service token #1679
• control-service: protect against v1 cron batch not being supported by @murphp15 in control-service: protect against v1 cron batch not being supported #1777
• docs: Documentation improvement: improve main README VDK images - gif by @zverulacis in Documentation improvement: improve main README VDK images - gif #1676
• docs: Fix CICD build for control service by @murphp15 in Fix CICD build for control service #1790
• docs: Hotfix: Update the URL for gif by @zverulacis in Hotfix: Update the URL for gif #1702
• docs: Improve documentation: before and after image by @zverulacis in Improve documentation: before and after image  #1696
• docs: Issue 1574: Update Contributing document by @gary-tai in Issue 1574: Update Contributing document #1741
• docs: Update VEP-1507 with CI/CD description by @DeltaMichael in docs: Update VEP-1507 with CI/CD description #1707
• docs: update contributor documentation by @DeltaMichael in docs/control-service: update contributor documentation #1779
• frontend: Create build and release jobs for vdk/shared package by @DeltaMichael in frontend: Create build and release jobs for vdk/shared package #1672
• frontend: Fix shebang in build_shared.sh by @DeltaMichael in frontend: Fix shebang in build_shared.sh #1743
• frontend: Rename vdk/shared to versatiledatakit/shared by @DeltaMichael in frontend: Rename vdk/shared to versatiledatakit/shared #1694
• frontend: bump data-pipelines dependency versions by @DeltaMichael in frontend: bump data-pipelines dependency versions #1706
• frontend: compile fix by @ivakoleva in frontend: compile fix #1697
• frontend: data-pipelines build and test running CI/CD by @ivakoleva in frontend: data-pipelines build and test running CI/CD #1671
• frontend: data-pipelines publish artifact CI/CD by @ivakoleva in frontend: data-pipelines publish artifact CI/CD #1675
• frontend: data-pipelines sync to freezepoint by @ivakoleva in frontend: data-pipelines sync to freezepoint #1746
• frontend: e2e tests (vdk: fix technical error #6) by @ivakoleva in frontend: e2e tests (#6) #1633
• frontend: e2e tests wiring and clientid by @ivakoleva in frontend: e2e tests wiring and clientid #1754
• frontend: gitlab ci variables for change locations by @ivakoleva in frontend: gitlab ci variables for change locations #1685
• frontend: package naming fixes by @ivakoleva in frontend: package naming fixes #1695
• specs: Add user journeys to VEP-1507 by @DeltaMichael in specs: Add user journeys to VEP-1507 #1750
• specs: VDK Enhancement Proposal for Metajobs (complete) by @yonitoo in specs: VDK Enhancement Proposal for Metajobs (complete) #1757
• specs: VEP-1493: Vault Integration For Secrets Storage by @dakodakov in VEP-1493: Vault Integration For Secrets Storage #1756
• specs: VEP-1739 Support for Multiple Python Versions by @doks5 in specs: VEP-1739 Support for Multiple Python Versions  #1748
• specs: edit vep-1507 for style and grammar by @DeltaMichael in specs: edit vep-1507 for style and grammar #1758
• specs: vep-1416: Change status of vep by @doks5 in [specs] vep-1416: Change status of vep #1721
• vdk-audit-plugin: expand forbidden events list by @mivanov1988 in vdk-audit-plugin: expand forbidden events list #1683
• vdk-heartbeat: fix test failures by @tozka in vdk-heartbeat: fix test failures #1703
• vdk-impala: Add optional parameter for staging table prefix by @sbuldeev in vdk-impala: Add optional parameter for staging table prefix #1666
• vdk-ipython: add finalize job by @duyguHsnHsn in vdk-ipython: add finalize job  #1744
• vdk-ipython: change the way job_input is introduced in a notebook by @duyguHsnHsn in vdk-ipython: change the way job_input is introduced in a notebook #1678
• vdk-jupyter: jupyter UI refactor code and handle react-dom version mismatch by @duyguHsnHsn in vdk-jupyter: jupyter UI refactor code and handle react-dom version mismatch #1727
• vdk-jupyter: Jupyter UI change the strategy on storing the data from user input by @duyguHsnHsn in vdk-jupyter: Jupyter UI change the strategy on storing the data from user input  #1624
• vdk-jupyter: add deploy job component to the UI by @duyguHsnHsn in vdk-jupyter: add deploy job component to the UI #1704
• vdk-jupyter: add py-to-ts-interfaces to build by @duyguHsnHsn in vdk-jupyter: add py-to-ts-interfaces to build #1765
• vdk-jupyter: load data before vdk operations by @duyguHsnHsn in vdk-jupyter: load data before vdk operations #1762
• vdk-jupyter: remove py-to-ts-interfaces because of build problems by @duyguHsnHsn in vdk-jupyter: remove py-to-ts-interfaces because of build problems #1691
• vdk-jupyter: rename python module by @duyguHsnHsn in vdk-jupyter: rename python module #1784
• vdk-meta-jobs: Introduce configuration module by @gageorgiev in vdk-meta-jobs: Introduce configuration module #1692
• vdk-meta-jobs: first version of the VEP Summary and Motivation sections by @yonitoo in vdk-meta-jobs: first version of the VEP Summary and Motivation sections #1698
• vdk-meta-jobs: implement a limit on starting jobs at once by @yonitoo in vdk-meta-jobs: implement a limit on starting jobs at once #1681
• vdk-meta-jobs: make _start_job() limit hit log more descriptive by @yonitoo in vdk-meta-jobs: make _start_job() limit hit log more descriptive #1771
• vdk-plugins: bump docker version and resuse common dind by @tozka in vdk-plugins: bump docker version and resuse common dind #1701
• vdk-plugins: generate test report by @tozka in vdk-plugins: generate test report #1732
• vdk-plugins: upgrade docker used by @tozka in vdk-plugins: upgrade docker used #1693
• vdk-quickstart: remove use of global varaibles in CI by @tozka in vdk-quickstart: remove use of global varaibles in CI #1726
• vdk-trino: stabilize vdk-trino tests by @tozka in vdk-trino: stabilize vdk-trino tests #1677
• versatile-data-kit: add hosts entry to gitlab runners by @ivakoleva in versatile-data-kit: add hosts entry to gitlab runners #1731
• versatile-data-kit: automate merging of dependabot PRs by @tozka in versatile-data-kit: automate merging of dependabot PRs #1725
• versatile-data-kit: dependabot auto-merge fix by @ivakoleva in versatile-data-kit: dependabot auto-merge fix #1688
• versatile-data-kit: mechanism for targetted PR notifications. by @tozka in versatile-data-kit: mechanism for targetted PR notifications. #1733
• versatile-data-kit: pre-commit hook for (S)CSS/JS/TS/HTML formatting by @ivakoleva in versatile-data-kit: pre-commit hook for (S)CSS/JS/TS/HTML formatting #1684
• versatile-data-kit: simplify release process by @tozka in versatile-data-kit: simplify release process #1668

New Contributors

• @yonitoo made their first contribution in vdk-meta-jobs: first version of the VEP Summary and Motivation sections #1698

Full Changelog: v0.11...v0.12

---

This discussion was created from the release Versatile Data Kit 0.12.