Build fail - tests

[AugustT]

I think that travis fails if a test tries to write to disk. I'm not 100% but it would be consistent with the current build failure in my PR

[goldingn]

The tests in your PR all work fine on my mac but fail on my ubuntu box, the same way Travis does.

This probably isn't the issue, but I'm confused by this bit at the top of testBuildModule.R:

dir <- tempdir()
dir.create(dir, showWarnings = FALSE)

I think dir is just be the name of the temp directory already in use, so the dir.create should do nothing (it would warn that the directory exists, but that's suppressed).

I guess it's possible this is freaking the travis environment out somehow

[goldingn]

Got it. Linux is case-sensitive in file paths, the other OSs aren't.

BuildModule writes as NewModule.R
and you're looking for NewModule.r

so it just needs minor edits in the tests

[goldingn]

Or rather that only fixes the first failure.

I suspect the rest have something to do with unlinking the temporary directory at the end of this BuildModule script, since the rest are to do with downloading. Will investigate further...

[AugustT]

Great, I came to the same conclusions. I forget that tempdir() returned a directory that existed. I thought it was a character vector to create a directory. Let's see...

[goldingn]

It doesn't seem to be just that, but will try on travis shortly to check

[goldingn]

Still failing: https://travis-ci.org/zoonproject/zoon/builds/84484148

I can replicate this when doing R CMD check at the console, but not when running interactively (or sourcing the test script). It passes fine interactively.

No helpful error messages either. Grrr.

[AugustT]

Thanks, yeah the PR fails. We fixed about half of them. These must be able to run since the last successful travis run included these tests... I just have to figure out what has changed since then.

[goldingn]

See here, which refers to a previous bug of mine:
r-lib/testthat#86 (comment)

I've just tested it by adding Sys.setenv("R_TESTS" = "") and also library(zoon) in test-all.R on my ubuntu box. No dice though, I still get the same errors in R CMD check

[goldingn]

These errors are consistent with an error in the occurrence module, and work2 being returned but containing only NULLs.

I can simulate it with this:

  work2 <- workflow(occurrence = SpOcc(species = 'Anopheles plumbeus',
                                       extent = c(-Inf, 10, 45, 65)),
                    covariate = UKAir,
                    process = OneHundredBackground,
                    model = RandomForest, 
                    output = SameTimePlaceMap)

So it looks like it's to do with downloading that data. Maybe the connection times out or doesn't have permission?

[AugustT]

Yeah, figures, but why did it work in the past and not now. The current (successfully building) version has the same test in it!

[AugustT]

...
package versions...
there is a new version of rgbif...
installing...

[AugustT]

Nope that still works

[AugustT]

Wait.... has the module changed between these commits??

[AugustT]

Nope not that either... argh

[goldingn]

weirdly I can't get bf7b272 to pass now either, and that's the last good build on Travis. That suggests (but it's not certain) that something external has changed - presumably spocc or one of its dependencies

[AugustT]

I also just ran that and found the same. Also I can replecate teh error on my machine using

devtools:::check()

[AugustT]

Okay, I'm going to comment these tests out and move on. I'll come back to it later by which time it might have fixed itself!

[goldingn]

Seems sensible.

I just ran the following code to find dependencies of spocc which have changed on CRAN in the last few days:

library(versions)
deps <- unique(names(get('imports', envir = spocc:::.__NAMESPACE__.)))
versions <- available.versions(deps[-1])  # first is base which we can't get versions for
idx <- sapply(versions, function (x) isTRUE(any(x$date >= as.Date('2015-10-07'))))
versions[idx]

$rgbif
   version       date available
1    0.8.9 2015-10-07      TRUE
2    0.8.8 2015-07-24      TRUE
3    0.8.6 2015-07-03      TRUE
4    0.8.4 2015-06-08      TRUE
5    0.8.0 2015-03-09      TRUE
6    0.7.7 2014-11-06      TRUE
7    0.7.0 2014-07-31     FALSE
8    0.6.2 2014-04-24     FALSE
9    0.6.0 2014-04-17     FALSE
10   0.5.0 2014-02-16     FALSE
11   0.4.0 2013-11-20     FALSE
12   0.3.0 2013-07-19     FALSE
13   0.2.0 2013-03-01     FALSE
14   0.1.5 2012-12-04     FALSE
15   0.0.1 2012-03-21     FALSE

Which would point the finger of blame squarely at rgbif.

But after running:

install.versions('rgbif', '0.8.8')

I still get the errors...

Maybe it's GBIF itself?

[goldingn]

Note to self: spocc's tests for occ use: skip_on_cran() which may help. I'm not clear from their yaml whether they check with the --as-cran flag.

devtools::check() checks as CRAN by default, so we haven't actually tried testing not-as-cran yet. It could be that GBIF is just a bit slow at the moment and check is killing off those connections before they finish.

Will try this and switching back to the metacran Travis yaml (with fix mentioned above) at some point.

[AugustT]

Found the problem. There was another unlink lurking in another test file. My bad.

[goldingn]

Aaaaah. Well done!

[AugustT]

This issue has returned, suggesting that the reason tests including spocc ran successfully for me was independent from the unlink line.

I have run check(cran = FALSE) and this passes correctly. This lends weight to your hypothesis about time outs.

It is interesting that there is a skip_on_cran() line in the spocc tests perhaps @sckott has had similar issues testing? - Scott we have some tests that use spocc to get species data. They run with as cran FALSE but not TRUE, any ideas? This issue seems to have appeared with the new version of rgbif.

I changed the tests to a much smaller extent, thinking that this might reduce download time and lessen the chance of time outs. I changed the extent from c(-10, 10, 45, 65) to c(-1, 1, 52, 53), but this had no effect on the outcome.

[AugustT]

I managed to get some console output from the tests and here is where it is going wrong:

package 'spocc' successfully unpacked and MD5 sums checked

  The downloaded binary packages are in
    C:\Users\tomaug\AppData\Local\Temp\RtmpoVxyyv\downloaded_packages
  Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) : 
    there is no package called 'MASS'

It looks like spocc uses MASS but this is not specified in the description. I can fix this issue now by loading in the MASS library prior to spocc in our module but perhaps this can be fixed in spocc?

[goldingn]

I'm not sure that's it. I just got this on my ubuntu box, using the version of SpOcc which loads MASS:

  Running ‘test-all.R’ [12s/23s]
 ERROR
Running the tests in ‘tests/test-all.R’ failed.
Last 13 lines of output:
  workSyn$report[[1]] inherits from NULL not list

  testthat results ================================================================
  OK: 145 SKIPPED: 0 FAILED: 6
  1. Failure (at testWholeWorkflows.R#122): Workflows with lists of modules work.
  2. Failure (at testWholeWorkflows.R#140): Workflows with lists of modules work.
  3. Failure (at testWholeWorkflows.R#282): workflow with mix of syntax works.
  4. Failure (at testWholeWorkflows.R#283): workflow with mix of syntax works.
  5. Failure (at testWholeWorkflows.R#284): workflow with mix of syntax works.
  6. Failure (at testWholeWorkflows.R#285): workflow with mix of syntax works.

  Error: testthat unit tests failed
  Execution halted

[AugustT]

Those are different failures...

It suggests you box is failing on this:

  workSyn <- workflow(occurrence = UKAnophelesPlumbeus,
                      covariate = 'UKAir',
                      process = BackgroundAndCrossvalid(k=2),
                      model = list(LogisticRegression, RandomForest),
                      output = Chain('SameTimePlaceMap', 'SameTimePlaceMap'))

[sckott]

hi all - so you think this is a spocc issue?

It is interesting that there is a skip_on_cran() line in the spocc tests ...

We do run tests nightly on Travis and Appveyor (passing now https://travis-ci.org/ropensci/spocc/builds/83326392). I just don't run on CRAN b/c well it's CRAN

[goldingn]

yeah, that fails on one ubuntu box and not the other, best we ignore that for now (though weird that it also fails with as-cran and not without).

[goldingn]

Using my otherwise working ubuntu box, the main repo version as of tom's previous PR: 323c6fd it works fine.

But if I uncomment those tests that depend on spocc ... it passes if I don't check as CRAN and fails if I do check as CRAN.

So I still think the main issue is some sort of CRAN testing timeout, rather than spocc...

[AugustT]

Hi @sckott ,

I was having issues calling occ2df in tests of the zoon package. The error in testing was:

 package 'spocc' successfully unpacked and MD5 sums checked

  The downloaded binary packages are in
    C:\Users\tomaug\AppData\Local\Temp\RtmpoVxyyv\downloaded_packages
  Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) : 
    there is no package called 'MASS'

Could this be because occ2df' callsMASS` functions which are not available when running as cran? By loading MASS before spocc on my PC I can fix this.

Probably not an issue to worry about since all your tests are passing.

@goldingn since I was getting an error there is no package called 'MASS', this seems a more likely explanation than time out i would have though

Tom

[goldingn]

@AugustT was this failing in test_that('modules downloading data work'... or elsewhere?

[sckott]

I'll have a look

[AugustT]

@goldingn Yes it was failing there. What I did was comment out all the tests and then uncomment L 57-66.

  work2 <- workflow(occurrence = SpOcc(species = 'Anopheles plumbeus',
                                       extent = c(-10, 10, 45, 65)),
                    covariate = UKAir,
                    process = OneHundredBackground,
                    model = RandomForest, 
                    output = SameTimePlaceMap)

  expect_true(exists('work2'))
  expect_equal(names(work2), expected_names) 
  expect_is(work2$occurrence.output[[1]], 'data.frame')

This meant that I got back more useful information in the console (because check() only shows you the last ~13 lines to console. This failed when MASS was not loaded and then passed when I added in the loading MASS line.

[goldingn]

Ah right. When I uncomment the whole thing, I get:

 ERROR
Running the tests in ‘tests/test-all.R’ failed.
Last 13 lines of output:
  work2$report[[1]] inherits from NULL not RasterLayer

  testthat results ================================================================
  OK: 158 SKIPPED: 0 FAILED: 6
  1. Failure (at testWholeWorkflows.R#67): modules downloading data work
  2. Failure (at testWholeWorkflows.R#68): modules downloading data work
  3. Failure (at testWholeWorkflows.R#74): modules downloading data work
  4. Failure (at testWholeWorkflows.R#75): modules downloading data work
  5. Failure (at testWholeWorkflows.R#76): modules downloading data work
  6. Failure (at testWholeWorkflows.R#77): modules downloading data work

So it looks like your MASS change fixed the first three lines but that there's still something else going on here as well.

[goldingn]

Sorry, miscounted lines. In fact I get a failure on line 67 that seems to work for you.

[sckott]

MASS isn't a dependency/import/suggest in spocc. I suppose it could be a dep of a spocc dep.

[sckott]

hmm, check and tests run fine for me locally. Where is the error occurring.

[goldingn]

I've just pushed a branch to my fork which should reproduce this: https://github.com/goldingn/zoon/tree/checkfail

and created a PR to force a Travis run: https://travis-ci.org/goldingn/zoon/builds/84933122

@sckott we have these tests commented out in the master, this fork is the failing one

[goldingn]

this all looks to be fixed now: https://travis-ci.org/goldingn/zoon/builds/85023423

[sckott]

glad its working. in my playing around I couldn't figure out anything wrong...

[goldingn]

Thanks for looking!

Will let you know of we figure out the root cause of that MASS/spocc error

[AugustT]

@goldingn Great work Nick!

Do I still need to use the 'skip on cran' command or are we simply going to let it fail on the cran test and be happy with this as long as it passes the other tests?

I also noticed that the Travis run now takes 1hr 30min. That seems like a very long time? Is that normal? My rnbn Travis takes only 1min 30sec, but that could be abnormally fast?

[goldingn]

Nope, it works without skip_on_cran on both now!

It should only take a little over 20 mins. There are four runs in parallel though and it reports the total.

1min 30s is impressive! Presumably its using docker on Travis. That's apparently tricky to do with a complex build like this, its an issue on r-builder though. The best way of speeding this up now would be reducing dependencies I think.

[AugustT]

Fab, great work for getting this sorted!