performance issue

[magomimmo]

When test task is used in a TDD workflow and you do not pass it any specific namespace, it's performance would not be judged acceptable by a TDD practitioner.

You can veryfy the issue as follows:

git clone https://github.com/magomimmo/modern-cljs.git
cd modern-cljs
git checkout se-tutorial-15
boot watch testing test -n modern-cljshopping.validators-test

Starting file watcher (CTRL-C to quit)...


Testing modern-cljs.shopping.validators-test

Ran 1 tests containing 13 assertions.
0 failures, 0 errors.
Elapsed time: 4.777 sec

then force a test failure as follows:

Modify the first unit test in the monder-cljs/test/cljc/modern_cljs/shopping/validators_test.cljc as follows:

(deftest validate-shopping-form-test
  (testing "Shopping Form Validation"
    (testing "/ Happy Path"
      (are [expected actual] (= expected actual)
           nil (validate-shopping-form "" "0" "0" "0")                 ;force a failure
           nil (validate-shopping-form "1" "0.0" "0.0" "0.0")
           nil (validate-shopping-form "100" "100.25" "8.25" "123.45")))

    ...))))

you'll see the following report:

Testing modern-cljs.shopping.validators-test

FAIL in (validate-shopping-form-test) (validators_test.cljc:9)
Shopping Form Validation / Happy Path
expected: nil
  actual: {:quantity
           ["Quantity can't be empty"
            "Quantity has to be an integer number"
            "Quantity can't be negative"]}
    diff: + {:quantity
             ["Quantity can't be empty"
              "Quantity has to be an integer number"
              "Quantity can't be negative"]}

Ran 1 tests containing 13 assertions.
1 failures, 0 errors.
clojure.lang.ExceptionInfo: Some tests failed or errored
    data: {:test 1, :pass 12, :fail 1, :error 0, :type :summary}
                clojure.core/ex-info       core.clj: 4593
   adzerk.boot-test/eval549/fn/fn/fn  boot_test.clj:   73
boot.task.built-in/fn/fn/fn/fn/fn/fn   built_in.clj:  233
   boot.task.built-in/fn/fn/fn/fn/fn   built_in.clj:  233
      boot.task.built-in/fn/fn/fn/fn   built_in.clj:  230
                 boot.core/run-tasks       core.clj:  701
                   boot.core/boot/fn       core.clj:  711
 clojure.core/binding-conveyor-fn/fn       core.clj: 1916
                                 ...
Elapsed time: 0.513 sec

As you see it takes 0.5 sec.

Now remove the forced failure:

(deftest validate-shopping-form-test
  (testing "Shopping Form Validation"
    (testing "/ Happy Path"
      (are [expected actual] (= expected actual)
           nil (validate-shopping-form "1" "0" "0" "0")                 ;fix the forced failure
           nil (validate-shopping-form "1" "0.0" "0.0" "0.0")
           nil (validate-shopping-form "100" "100.25" "8.25" "123.45")))

    ...))))

As you see it takes only 0.3 sec to get the results again:

Testing modern-cljs.shopping.validators-test

Ran 1 tests containing 13 assertions.
0 failures, 0 errors.
Elapsed time: 0.327 sec

Stop boot process and relaunch it without specifying any namespace:

boot watch testing test

Starting file watcher (CTRL-C to quit)...


Testing modern-cljs.core

Testing modern-cljs.login

Testing modern-cljs.login.validators

Testing modern-cljs.remotes

Testing modern-cljs.shopping.validators

Testing modern-cljs.shopping.validators-test

Testing modern-cljs.templates.shopping

Ran 1 tests containing 13 assertions.
0 failures, 0 errors.
Elapsed time: 8.998 sec

Now it takes 9 sec, instead of 4 sec to launch the tests. Force a failure as before.

Testing modern-cljs.core

Testing modern-cljs.login

Testing modern-cljs.login.validators

Testing modern-cljs.remotes

Testing modern-cljs.shopping.validators

Testing modern-cljs.shopping.validators-test

FAIL in (validate-shopping-form-test) (validators_test.cljc:9)
Shopping Form Validation / Happy Path
expected: nil
  actual: {:quantity
           ["Quantity can't be empty"
            "Quantity has to be an integer number"
            "Quantity can't be negative"]}
    diff: + {:quantity
             ["Quantity can't be empty"
              "Quantity has to be an integer number"
              "Quantity can't be negative"]}

Testing modern-cljs.templates.shopping

Ran 1 tests containing 13 assertions.
1 failures, 0 errors.
clojure.lang.ExceptionInfo: Some tests failed or errored
    data: {:test 1, :pass 12, :fail 1, :error 0, :type :summary}
                clojure.core/ex-info       core.clj: 4593
   adzerk.boot-test/eval549/fn/fn/fn  boot_test.clj:   73
boot.task.built-in/fn/fn/fn/fn/fn/fn   built_in.clj:  233
   boot.task.built-in/fn/fn/fn/fn/fn   built_in.clj:  233
      boot.task.built-in/fn/fn/fn/fn   built_in.clj:  230
                 boot.core/run-tasks       core.clj:  701
                   boot.core/boot/fn       core.clj:  711
 clojure.core/binding-conveyor-fn/fn       core.clj: 1916
                                 ...
Elapsed time: 5.868 sec

Now it takes almost 6 sec to rerun the tests. Fix the forced failure

Testing modern-cljs.core

Testing modern-cljs.login

Testing modern-cljs.login.validators

Testing modern-cljs.remotes

Testing modern-cljs.shopping.validators

Testing modern-cljs.shopping.validators-test

Testing modern-cljs.templates.shopping

Ran 1 tests containing 13 assertions.
0 failures, 0 errors.
Elapsed time: 5.358 sec

as you see it takes again more than 5 sec to rerun the tests.

Is that normal?

[Deraen]

This is because boot-test runs tests in a new pod each time. This means that all namespaces need to be compiled each time, instead only the changed ones.

Alternative (used by e.g. Midje) would be to use tools.namespace to reload changed namespaces in the same pod each time.

[arichiardi]

About this, can we actually do anything about it? It would be really great if boot had a good story for tests. I know that @Deraen has developed an alternative (boot-alt-test) but they both feel incomplete at the moment and it would be awesome to focus our energy on either one or the other. Lein beats us here and we don't want that 😈