simphony

a bare metal orchestrator made with python and rabbit and reddis

python -> to write code
rabbit -> to maintain queues
reddis -> to maintain states

How to use it:

write loose functions in dags/funcs.py
Define it as Make(funcA) >> Make(funcB) to make link
define your dag in controller.py
run controller.py in one shell
run worker.py in as many shells you want

TODO:

connector for redis
auto push pull from redis using task ids using decorators
save dag_ids and task_ids in db
write connectors for sqlite,mysql,postgres etc
keep flag of STATELESS to skip all DB operations and use redis only
multitreading in workers (concurrency)
multiprocessing in workers (parallelization)
Have controller in class ✅
FastAPI to trigger dag controller
use task definations and task flow from same file ✅
UI ? idk