Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

- implement hooks #826

Open
wants to merge 4 commits into
base: development
Choose a base branch
from
Open

- implement hooks #826

wants to merge 4 commits into from

Conversation

mk01
Copy link
Contributor

@mk01 mk01 commented Jul 3, 2021

  • enable user to extend plotman's functionality / plotting flow,
    by providing external scripts, without need to touch plotmans
    internals

  • many of current feature requests can be solved with lighweight
    scripts (bash/python/...), e.g. Start Rsync automatically when Plotman starts #712, Clean temp dirs if no plos are running #711, Automatically suspend jobs on stage 3:6 if not enought space in dst disk #677, Stop multiple plots copying over to the HDD at the same time.  #638, plotman kill doesn't remove all temp files #582 ...

  • process cpu and io niceness can also be manipulated, affinity
    to cpu or NUMA on big server hardware can be set differently
    based on stage or plotter

  • all new code is in new library 'hooks' + two new shell scripts
    directly inside hooks.d directory (located inside plotman's
    own config directory

  • supplied hook serves as reference implementation / example,
    providing functionality for Automatically suspend jobs on stage 3:6 if not enought space in dst disk #677

  • modification of existing plotman's code is only to call entrypoint
    in hooks.py and pass current jobs list containing jobs objects

  • currently manager.py maybe_start_new_plot() and plotman.py kill
    is injected with call into hooks.try_run() and hooks.run() respectively

  • try_run consumes fully refreshed jobs[], compares phase to previous
    job phase and if it is changed, calls hooks.run()

  • run() takes plotmans environment, extends it with particular job's
    metadata and calls all executable files from hooks.d directory
    having extension .sh or .py

  • scripts are called synchronously, cmd exec is waiting until the
    script process returns. that means the implementation is not suitable
    for LONG running actions.

  • anyhow, plotman CLI can be called without issues from within the hooks,
    recursion in job_refresh -> try_run -> hooks script -> plotman cmd
    is being checked for

  • stdout/stderr of running the hook scripts is currently forced to /tmp/plotman-hooks.log

2021-07-03T20:45:29 [01-check-dstdir-free.sh(2fe2355c)] INFO: processing plot id 2fe2355c8e6e7faf21e1aec895f0167c7e72054a5e3d39ddfee854193c7cd00c (ph: 1:5)
2021-07-03T20:45:29 [01-check-dstdir-free.sh(2fe2355c)] INFO: plot not at phase 3:6/7, hook done
2021-07-03T20:45:50 [01-check-dstdir-free.sh(480fd668)] INFO: processing plot id 480fd6680080bf018a532940b442a27565ab0c9616e2f7900aad6423a486899e (ph: 3:7)
2021-07-03T20:45:50 [01-check-dstdir-free.sh(480fd668)] INFO: there is 1386222714880 available space at /chia/013/, which is > k32plot 108984795136. job can continue.

new: src/plotman/hooks.py
src/plotman/resources/hooks.d

update: src/plotman/configuration.py
src/plotman/manager.py
setup.cfg

@mk01 mk01 force-pushed the hooks branch 2 times, most recently from 05b5ff0 to 97d901c Compare July 3, 2021 21:13
mk01 added 4 commits July 4, 2021 11:47
- enable user to extend plotman's functionality / plotting flow,
  by providing external scripts, without need to touch plotmans
  internals
- many of current feature requests can be solved with lighweight
  scripts (bash/python/...), e.g. ericaltendorf#712, ericaltendorf#711, ericaltendorf#677, ericaltendorf#638, ericaltendorf#582
  ...,
- process cpu and io niceness can also be manipulated, affinity
  to cpu or NUMA on big server hardware can be set differently
  based on stage or plotter

- all new code is in new library 'hooks' + two new shell scripts
  directly inside hooks.d directory (located inside plotman's
  own config directory
- supplied hook serves as reference implementation / example,
  providing functionality for ericaltendorf#677
- modification of existing plotman's code is only to call entrypoint
  in hooks.py and pass current jobs list containing jobs objects

- currently manager.py maybe_start_new_plot() and plotman.py kill
  is injected with call into hooks.try_run() and hooks.run() respectively
- try_run consumes fully refreshed jobs[], compares phase to previous
  job phase and if it is changed, calls hooks.run()
- run() takes plotmans environment, extends it with particular job's
  metadata and calls all executable files from hooks.d directory
  having extension .sh or .py

- scripts are called synchronously, cmd exec is waiting until the
  script process returns. that means the implementation is not suitable
  for LONG running actions.
- anyhow, plotman CLI can be called without issues from within the hooks,
  recursion in job_refresh -> try_run -> hooks script -> plotman cmd
  is being checked for

new:	src/plotman/hooks.py
	src/plotman/resources/hooks.d

update: src/plotman/configuration.py
	src/plotman/manager.py
	setup.cfg
  hook scripts run
	- to prevent expansion of paths / params to simple *
	  e.g. $PLOTMAN_DSTDIR/*$PLOTMAN_PLOTID* -> /*
          if $VARS would be missing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant