Jobdef hooks

While you can include arbitrary Clojure code anywhere, use hooks for a more structured approach to executing actions as part of your job. Hooks are simply functions that can be executed before and/or after the main fire! action.

To be clear, "after fire!" means after the fire! function returns. I.e. after the EMR launch request has been sent (NOT when the job completes). BUT, in the case of local mode, fire! does return AFTER the hadoop job completes. If you want to do something after the cluster starts or after your Step completes, see 'wait' and 'wait-on-step' in examples/sample-jobdef.clj.

Common uses might be to validate the results of a local test, to download some files locally before execution starts, or to modify an input file before it is used.

Add hooks anytime before fire! is called, using the add-hooks function:

  (add-hooks ([optional-conditional-expr] function-to-execute)* )

It takes zero or more functions, each one optionally preceded by a conditional expression. If there is a conditional expression, the fn following it is only added if the expression is true.

The function-to-execute can be a 1-arg or 2-arg fn or have both 1-arg and 2-arg signatures (see defn at http://clojure.org/functional_programming for details on how to make a fn with multiple arities).

The 1-arg signatures are called, in the order they are added, before fire! I.e. these are "pre hooks". The arg value is:

  1. eopts: the options map

The 2-arg signatures are called, in reverse order, after fire! I.e. these are "post hooks". The argument values are:

  1. eopts: The options map
  2. state: the result of the 1-arg call for the same function. If there is no
            1-arg signature for this fn, then nil.

For example:

  (defn- say-hello [eopts] (println "Hello"))

  (add-hooks
    (or (run?) (local?)) say-hello
    (fn [eopts _] (println "goodbye")))

As you can see, either named functions or anonymous functions work. This example will print "Hello" when your job is started with 'lemur run' or 'lemur local', and in all cases, it will print "goodbye" afterward.

Another example, using a fn with both 1-arg and 2-arg signatures, with state:

  (defn- greet
    ([eopts] (System/currentTimeMillis))
    ([eopts state] (println "fire! time in millis: " (- (System/currentTimeMillis) state))))

  (add-hooks
    greet)

No conditional expression was provided, so the greet hook is always added. It has both a 1-arg and a 2-arg signature. The 1-arg is triggered before fire!, and the 2-arg is triggered after (and is given the result of the 1-arg call as it's second argument).

Recommendation: Have your fn test for (dry-run?) and print some useful diagnostic output in this case. At some point, I may add more semantics around these hook functions for collecting diagnostic info.