Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Workflow module implementation #276

Draft
wants to merge 167 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
167 commits
Select commit Hold shift + click to select a range
abb75fb
chore: remove jinja2
0hsn Oct 5, 2023
35aac91
chore: add better debugging tool
0hsn Oct 5, 2023
5687eba
chore: add pydentic pkg
0hsn Oct 5, 2023
abe5f26
feat: reusable test fixtures
0hsn Oct 7, 2023
cf1adf7
feat: workflow document with test
0hsn Oct 7, 2023
3bcc479
feat: workflow driver with tests
0hsn Oct 7, 2023
d16d23b
refactor: update file name parse logic
0hsn Oct 8, 2023
4f1835e
feat: entity for internal chkware tasks
0hsn Oct 8, 2023
9857a43
feat: entity for internal chkware validate task
0hsn Oct 8, 2023
8194651
feat: parsing tasks in workflow
0hsn Oct 8, 2023
9421e41
refactor: remove unused
0hsn Oct 8, 2023
fc68e70
chore: package version update
0hsn Oct 12, 2023
9fdec94
refactor: versioned document v2 moved to core document
0hsn Oct 14, 2023
e580f67
feat: support versioned document v2 in VariableTableManager.handle
0hsn Oct 14, 2023
07d9483
feat: process task signature
0hsn Oct 14, 2023
0752909
chore: update packages
0hsn Oct 29, 2023
f0d030f
feat: disallow extra data in input yaml
0hsn Oct 29, 2023
bec7f22
feat: move ParsedTask to entities
0hsn Oct 29, 2023
c165129
refactor: use __future__.annotations
0hsn Oct 29, 2023
ee839be
fix: add missing field
0hsn Oct 29, 2023
df6df40
feat: use parsed task instead of dict
0hsn Oct 29, 2023
ebc72f6
refactor: move all domain code to main module
0hsn Nov 1, 2023
7e88ee2
feat: add ExecResponse to capture generic response
0hsn Nov 2, 2023
ece5fe5
feat: add call api to fetch module
0hsn Nov 2, 2023
5be3521
refactor: type isolation on replace_value
0hsn Nov 2, 2023
c54e9f3
feat: tests for fetch.call
0hsn Nov 2, 2023
dc68bd9
refactor: convert static method to class method for VariableTableManager
0hsn Nov 3, 2023
9724d3c
refactor: move loading envvars in handle_environment
0hsn Nov 3, 2023
813c16a
refactor: reflect new api change from platform
0hsn Nov 3, 2023
3fb7326
feat: add slugify helper method
0hsn Nov 3, 2023
5d29fcd
feat: type of workflow supported stirngs
0hsn Nov 3, 2023
f79a8d3
feat: support name and id for workflow
0hsn Nov 3, 2023
70f8345
feat: add name and id to workflow doc
0hsn Nov 3, 2023
158b64a
feat: initial workflow run for fetch done
0hsn Nov 3, 2023
a6ad3e3
fix: tests for WorkflowDocument
0hsn Nov 5, 2023
ab95731
fix: remove load env test for combine_initial_variables
0hsn Nov 5, 2023
aa5fc44
feat: remove unused param doc
0hsn Nov 5, 2023
d66e77a
fix: update json data struct as the API changed
0hsn Nov 5, 2023
bbf4b85
feat: use class signature from return
0hsn Nov 5, 2023
5d4871d
feat: add extra data field for ExecResponse
0hsn Nov 5, 2023
045d808
feat: call function in validate module
0hsn Nov 5, 2023
28f8e28
feat: simple data validation spec
0hsn Nov 5, 2023
a58ada5
feat: dumping output for workflow
0hsn Nov 5, 2023
4a81197
chore: update packages
0hsn Nov 15, 2023
519aabd
refactor: update to avoid slow regular expressions is security-sensitive
0hsn Nov 15, 2023
2fcb5a4
chore: package version update
0hsn Nov 21, 2023
ab9be76
feat: pass dict data from context
0hsn Nov 21, 2023
a51b759
refactor: separation of data parsing
0hsn Nov 28, 2023
484acc4
chore: pkg version update after conflict resolve
0hsn Dec 24, 2023
85ff63a
chore: package version update after rebase
0hsn May 1, 2024
03ee532
chore: package update 19.MAY.2024
0hsn May 19, 2024
a1194dc
chore: package version update
0hsn Jun 1, 2024
372c81b
chore: remove jinja2
0hsn Oct 5, 2023
0561e58
chore: add better debugging tool
0hsn Oct 5, 2023
2e3be05
chore: package version update
0hsn Oct 12, 2023
872cd6e
chore: update packages
0hsn Nov 15, 2023
fdbd19c
chore: package version update after rebase
0hsn May 1, 2024
9e98eab
chore: package update 19.MAY.2024
0hsn May 19, 2024
74a7d5c
chore: package version update
0hsn Oct 12, 2023
572f215
chore: package version update
0hsn Nov 21, 2023
d0a749f
chore: pkg version update after conflict resolve
0hsn Dec 24, 2023
bc64bd6
chore: package version update after rebase
0hsn May 1, 2024
c6b9015
chore: package version update
0hsn Oct 12, 2023
6edd9f6
chore: update packages
0hsn Nov 15, 2023
73019f0
chore: package version update
0hsn Nov 21, 2023
6d7e326
chore: pkg version update after conflict resolve
0hsn Dec 24, 2023
b1b2426
feat: add property to get file path and dir path from file context
0hsn Dec 28, 2023
9fe219c
feat: make base out of dir support for PathFrom
0hsn Dec 28, 2023
069b551
chore: update package version
0hsn May 1, 2024
efba3b5
feat: update packages
0hsn May 19, 2024
fe23a53
feat: update signature for after_hook
0hsn May 19, 2024
074c9d4
chore: install hence and update packages
0hsn May 26, 2024
e49f701
refactor: naming update
0hsn May 26, 2024
94ecf0c
chore: package version update
0hsn Jun 1, 2024
83f9fdc
chore: package updates
0hsn Jul 9, 2024
3bd8f7e
feat: hence version 0.9.0 update
0hsn Jul 9, 2024
2ec9041
chore: fix hence update
0hsn Jul 10, 2024
d0ef301
feat: helper for finding absolute file path
0hsn Jul 10, 2024
89c48ff
refactor: remove PathFrom
0hsn Jul 10, 2024
a28ddb2
feat: define task domain object
0hsn Jul 10, 2024
56b910a
refactor: code format update
0hsn Jul 10, 2024
4a5ceca
feat: implement fetch task
0hsn Jul 10, 2024
5f9c545
feat: add ValidationTask domain class
0hsn Jul 10, 2024
e407fa6
refactor: general code linting fix
0hsn Jul 10, 2024
e77e4c1
feat: create validation task
0hsn Jul 10, 2024
bc4ad25
feat: implement task based workflow
0hsn Jul 10, 2024
cdd0a6b
feat: update domain models for validation tasks
0hsn Jul 10, 2024
de5165a
chore: package version update
0hsn Jul 17, 2024
10ccaa3
feat: remove ParsedTask and related utils
0hsn Jul 18, 2024
8444c72
feat: add run_id to ChkwareTask
0hsn Jul 18, 2024
90461c1
chore: hence version update with other package update
0hsn Aug 24, 2024
7cbbf2f
chore: package version update
0hsn Aug 26, 2024
b84a6fc
refactor: update title keys for updated hence version
0hsn Aug 26, 2024
a475753
feat: task running with success and get result
0hsn Aug 26, 2024
8ab10d9
feat: print overall result
0hsn Aug 27, 2024
3bdea59
Merge pull request #289 from 0hsn/feat/module-workflow-1
0hsn Aug 27, 2024
bdc392a
feat: more test spec added
0hsn Aug 27, 2024
d4060bb
feat: workflow runs as expected for simple workflow
0hsn Aug 28, 2024
0e3c07c
refactor: add url params as separate section
0hsn Aug 29, 2024
136ede7
refactor: to make similar sign as other module
0hsn Aug 29, 2024
75f0434
refactor: handling obj init in contractor
0hsn Aug 31, 2024
a54888e
feat: handle absolute path conversion in ChkwareTask
0hsn Aug 31, 2024
1f26da6
docs: added todo for next update
0hsn Sep 1, 2024
d75bf85
refactor: improve checking
0hsn Sep 2, 2024
c551b83
refactor: move making task to separate function
0hsn Sep 2, 2024
1df2291
chore: remove hence
0hsn Sep 3, 2024
e43c75b
refactor: remove hence usages
0hsn Sep 3, 2024
922bd59
feat: pass variable from wf to validation
0hsn Sep 5, 2024
63d8248
refactor: formatting fix
0hsn Sep 6, 2024
a011ac6
feat: expose data added for wf spec
0hsn Sep 6, 2024
de5591d
feat: chkware task validation
0hsn Sep 6, 2024
fae11ba
feat: enum and task params entity
0hsn Sep 6, 2024
7a52ad6
feat: support for VersionedDocumentV2 on ExposeManager
0hsn Sep 7, 2024
d04a072
refactor: remove ExposableVariables, use Variables instead
0hsn Sep 8, 2024
8aea013
feat: add extra data by category in ExecResponse
0hsn Sep 8, 2024
508905e
feat: new doc version and str dunder added
0hsn Sep 8, 2024
a6b3ec6
feat: return all steps
0hsn Sep 8, 2024
f10a84e
feat: workflow run happy path success
0hsn Sep 8, 2024
13f34ae
refactor: reorder import
0hsn Sep 17, 2024
d69a0f8
feat: move presentation to separate display class
0hsn Sep 17, 2024
2f6110a
feat: add step exposed data to response
0hsn Sep 17, 2024
7bec938
feat: pass step response as list of exec response
0hsn Sep 17, 2024
6a2549f
feat: impl presentation service
0hsn Sep 18, 2024
5e501eb
refactor: normalize code
0hsn Sep 18, 2024
c748fb6
refactor: import reorder
0hsn Sep 18, 2024
fadb0ef
feat: sent fetch expection in exec response
0hsn Sep 18, 2024
b186de0
feat: add exposed data to exec response
0hsn Sep 18, 2024
8435eac
feat: add exposed data to exec response
0hsn Sep 18, 2024
39aaae5
fix: dumping response for validate
0hsn Sep 19, 2024
459a8bf
feat: add report to validate exec response
0hsn Sep 19, 2024
645dc85
feat: save step results
0hsn Sep 19, 2024
ea56e93
feat: pass presentation items
0hsn Sep 20, 2024
0c53a4d
feat: dump str final
0hsn Sep 20, 2024
c83e77f
feat: print displaying wf response
0hsn Sep 21, 2024
ddd8eb0
feat: get_exposed_replaced_data v2
0hsn Sep 23, 2024
6c95b62
refactor: instead of passed task info, pass the task
0hsn Sep 23, 2024
ca09d77
feat: try to covert to dict service method
0hsn Sep 27, 2024
36528f5
feat: update messaging for list for exportables
0hsn Sep 27, 2024
d700be9
refactor: reformation of spacing in output
0hsn Sep 28, 2024
4d5a6cd
feat: using the value of the enum
0hsn Sep 29, 2024
68947cc
refactor: remoce dumping
0hsn Sep 30, 2024
f53acfa
refactor: change fn sign
0hsn Sep 30, 2024
a019696
fix: json object dumping iissue
0hsn Oct 2, 2024
dae7fb8
feat: convert validation doc to v2
0hsn Oct 3, 2024
557dd60
feat: replace dataclass with basemodel
0hsn Oct 3, 2024
d556d69
docs: add feature suggestion
0hsn Oct 3, 2024
a1e95d0
feat: remove templating notation
0hsn Oct 3, 2024
55d6dea
feat: exposing dict based data in validation
0hsn Oct 3, 2024
c2f251c
feat: convert dataclass to basemodel based doc
0hsn Oct 3, 2024
b7ec614
feat: use replace utils ver2
0hsn Oct 3, 2024
e28c993
chore: package version update
0hsn Oct 4, 2024
eae1fd8
feat: add duct converted value
0hsn Oct 4, 2024
be757b6
feat: pass variables
0hsn Oct 4, 2024
f0dcd56
feat: use exposed value for dumping
0hsn Oct 4, 2024
a5c7196
refactor: remove unused func
0hsn Oct 4, 2024
f63bdeb
refactor: remove output data in exposed data uses
0hsn Oct 4, 2024
c6a7e9a
feat: finalize formatted and json dump for wf
0hsn Oct 4, 2024
5adddcc
fix: document type change in test
0hsn Oct 4, 2024
5968179
fix: issues with object type change
0hsn Oct 4, 2024
353fa49
feat: return inner data
0hsn Oct 4, 2024
d288434
fix: tests for wf tasks and services
0hsn Oct 4, 2024
767251f
fix: path issue caught on ci
0hsn Oct 4, 2024
157f9cd
fix: OS neutral path
0hsn Oct 4, 2024
c1642d9
refactor: from str concat to path concat
0hsn Oct 4, 2024
78b9549
refactor: reorder imports
0hsn Oct 4, 2024
8acf3a0
refactor: update paths to OS deps
0hsn Oct 4, 2024
e80284c
Merge pull request #312 from 0hsn/feat/module-workflow-2
0hsn Oct 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,20 @@ requests = "*"
cerberus = "*"
defusedxml = "*"
xmltodict = "*"
jinja2 = "*"
python-dotenv = "*"
pydantic = "*"

[dev-packages]
pytest = "*"
mypy = "*"
pylint = "*"
flake8 = "*"
black = {extras = ["d"], version = "*"}
var-dump = "*"
types-pyyaml = "*"
types-requests = "*"
pytest-cov = "*"
coveralls = "*"
icecream = "*"

[requires]
python_version = "3.11"
1,250 changes: 692 additions & 558 deletions Pipfile.lock

Large diffs are not rendered by default.

8 changes: 2 additions & 6 deletions chk/console/main.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
"""Commands"""

import typing
from os import environ

import click
from dotenv import load_dotenv

import chk.modules.fetch as fetch_executor
import chk.modules.validate as validate_executor
Expand All @@ -30,12 +28,10 @@ def combine_initial_variables(external_vars: str, **kwargs: typing.Any) -> dict:
"""Reads a json string and converts to dict, and combines with env and dotenv
variables"""

load_dotenv()
return load_variables_as_dict(external_vars, **kwargs)

return load_variables_as_dict(external_vars, **kwargs) | {"_ENV": dict(environ)}


def after_hook(resp: object) -> None:
def after_hook(resp: dict) -> None:
"""Saves custom data from commands to global context bus

Args:
Expand Down
15 changes: 14 additions & 1 deletion chk/infrastructure/document.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
"""
Base document and utility
"""

import dataclasses

import cerberus
from pydantic import BaseModel, Field

from chk.infrastructure.file_loader import FileContext

Expand All @@ -18,11 +20,22 @@ class VersionedDocument:
version: str = dataclasses.field(default_factory=str)


class VersionedDocumentV2(BaseModel):
"""
versioned document entity
"""

context: tuple = Field(default_factory=tuple)
version: str = Field(default_factory=str)


class VersionedDocumentSupport:
"""DocumentVersionSupport"""

@staticmethod
def validate_with_schema(schema: dict, doc: VersionedDocument) -> bool:
def validate_with_schema(
schema: dict, doc: VersionedDocument | VersionedDocumentV2
) -> bool:
"""Validate a document with given schema

Args:
Expand Down
60 changes: 40 additions & 20 deletions chk/infrastructure/file_loader.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
"""
File loader utility
"""

from __future__ import annotations

import hashlib
from typing import NamedTuple
import json
from pathlib import Path
from typing import NamedTuple

import json
import yaml

from chk.infrastructure.typing_extras import JsonDecodingError
Expand Down Expand Up @@ -81,7 +84,7 @@ class FileContext(NamedTuple):
filepath_hash: str = ""

@staticmethod
def from_file(file: str, **kwarg: dict) -> "FileContext":
def from_file(file: str, **kwarg: dict) -> FileContext:
FileLoader.is_file_ok(file)
absolute_path = str(Path(file).absolute())
fpath_hash = hashlib.sha256(absolute_path.encode("utf-8")).hexdigest()
Expand All @@ -95,31 +98,48 @@ def from_file(file: str, **kwarg: dict) -> "FileContext":
arguments=kwarg["arguments"] if "arguments" in kwarg else {},
)

@property
def filepath_as_path(self) -> Path:
"""Get filepath as Path"""

class PathFrom:
"""Utility to expand to full path"""
return Path(self.filepath)

def __init__(self, base: Path):
self.base = base.absolute().parent
@property
def filepath_base_as_path(self) -> Path:
"""Get filepath parent or base as Path"""

def absolute(self, target: str) -> str:
"""Find absolute in comparison to base URL"""
return Path(self.filepath).absolute().parent

if target.startswith("./") or target.startswith("../"):
if self.base.exists():
to_path = self.base

target_path_sp = target.split("/")
for part in target_path_sp:
if part == "..":
to_path = to_path.parent
else:
to_path = Path(str(to_path) + "/" + part)
def generate_abs_path(base_: str, target_: str) -> str:
"""Generate absolute path in comparison to base path
Args:
base_: str, base path to calculate from
target_: str, file path that need absolute path

return str(to_path)
raise ValueError("Invalid base path.")
Returns:
Absolute path for given filepath
"""

base = Path(base_)
base_abs = base.absolute().parent if base.is_file() else base.absolute()

if not base_abs.exists():
raise ValueError("Invalid base path.")
if not (target_.startswith("./") or target_.startswith("../")):
raise ValueError("Invalid target path.")

to_path = base_abs
target_path_sp = target_.split("/")

for part in target_path_sp:
if part == "..":
to_path = to_path.parent
else:
to_path = to_path / part

return str(to_path)


class ExecuteContext(NamedTuple):
"""Information storage for execution context"""
Expand Down
25 changes: 25 additions & 0 deletions chk/infrastructure/helper.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
"""
Helper functions module
"""

import ast
import re
from collections.abc import Callable
from typing import Any

Expand Down Expand Up @@ -157,6 +159,16 @@ def to_auto(var: str) -> Any:

return var

@staticmethod
def try_dict(to_dict: Any, say_exception: bool = False) -> dict | Any:
try:
return dict(to_dict)
except ValueError as ex:
if say_exception:
raise ex
else:
return to_dict


def formatter(message: object, cb: Callable = str, dump: bool = True) -> str:
"""Format message with given callback
Expand All @@ -175,3 +187,16 @@ def formatter(message: object, cb: Callable = str, dump: bool = True) -> str:
click.echo(printable)

return printable


def slugify(string: str) -> str:
"""Make slug out of string"""

if not isinstance(string, str):
raise TypeError("slugify: only string value supported.")

string = string.lower().strip()
string = re.sub(r"[^\w\s-]", "", string)
string = re.sub(r"[\s_-]+", "-", string)
string = re.sub(r"-{2,}", "", string)
return string
Loading