Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

- implement hooks #826

Open
wants to merge 4 commits into
base: development
Choose a base branch
from
Open

- implement hooks #826

wants to merge 4 commits into from

Commits on Jul 4, 2021

  1. - implement hooks 

    - enable user to extend plotman's functionality / plotting flow,
  by providing external scripts, without need to touch plotmans
  internals
- many of current feature requests can be solved with lighweight
  scripts (bash/python/...), e.g. ericaltendorf#712, ericaltendorf#711, ericaltendorf#677, ericaltendorf#638, ericaltendorf#582
  ...,
- process cpu and io niceness can also be manipulated, affinity
  to cpu or NUMA on big server hardware can be set differently
  based on stage or plotter

- all new code is in new library 'hooks' + two new shell scripts
  directly inside hooks.d directory (located inside plotman's
  own config directory
- supplied hook serves as reference implementation / example,
  providing functionality for ericaltendorf#677
- modification of existing plotman's code is only to call entrypoint
  in hooks.py and pass current jobs list containing jobs objects

- currently manager.py maybe_start_new_plot() and plotman.py kill
  is injected with call into hooks.try_run() and hooks.run() respectively
- try_run consumes fully refreshed jobs[], compares phase to previous
  job phase and if it is changed, calls hooks.run()
- run() takes plotmans environment, extends it with particular job's
  metadata and calls all executable files from hooks.d directory
  having extension .sh or .py

- scripts are called synchronously, cmd exec is waiting until the
  script process returns. that means the implementation is not suitable
  for LONG running actions.
- anyhow, plotman CLI can be called without issues from within the hooks,
  recursion in job_refresh -> try_run -> hooks script -> plotman cmd
  is being checked for

new:	src/plotman/hooks.py
	src/plotman/resources/hooks.d

update: src/plotman/configuration.py
	src/plotman/manager.py
	setup.cfg
    @mk01
    mk01 committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    951e4b1 View commit details
    Browse the repository at this point in the history

  2. - add some sanitization / checks to ENV variables needed for proper 

      hook scripts run
	- to prevent expansion of paths / params to simple *
	  e.g. $PLOTMAN_DSTDIR/*$PLOTMAN_PLOTID* -> /*
          if $VARS would be missing
    @mk01
    mk01 committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    1dd5ca4 View commit details
    Browse the repository at this point in the history

  3. - add simple "hello world" kind of hook which logs new plotting process

    @mk01
    mk01 committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    f8e7880 View commit details
    Browse the repository at this point in the history

  4. - implement post kill cleanup based on plotid wildcard search 

      through dst/tmp/tmp2 dirs (ericaltendorf#582,...)
    @mk01
    mk01 committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    82e22b6 View commit details
    Browse the repository at this point in the history