Devel convergence vignette #53

be-green · 2020-01-25T17:26:33Z

Adds a vignette that helps people address and understand convergence warnings, with a brief, high-level intro to MCMC sampling and HMC to give context for why they are occurring. If this isn't useful or you have thoughts on the content (maybe I did something wrong or misrepresented something) please let me know.

codecov-io · 2020-01-25T17:50:22Z

Codecov Report

Merging #53 into devel will decrease coverage by 1.41%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            devel      #53      +/-   ##
==========================================
- Coverage   65.64%   64.22%   -1.42%     
==========================================
  Files          26       25       -1     
  Lines        4590     4048     -542     
==========================================
- Hits         3013     2600     -413     
+ Misses       1577     1448     -129

Impacted Files	Coverage Δ
R/baggr.R	`93.33% <100%> (-0.42%)`	⬇️
R/mint.R	`86.95% <0%> (-4.35%)`	⬇️
R/baggr_plot.R	`93.75% <0%> (-3.13%)`	⬇️
R/auto_prior.R	`87.2% <0%> (-3.12%)`	⬇️
R/group_effects.R	`84.84% <0%> (-2.25%)`	⬇️
src/stan_files/rubin.hpp	`81.2% <0%> (-1.66%)`	⬇️
src/stan_files/logit.hpp	`80.43% <0%> (-1.23%)`	⬇️
src/stan_files/mutau.hpp	`82.38% <0%> (-0.44%)`	⬇️
R/fixed_effects.R
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dedf373...a388df2. Read the comment docs.

wwiecek · 2020-01-26T14:58:09Z

I will try to review ASAP, but I have a few urgent things to work on this week. We've been discussing a vignette where we show how to do MCMC/Bayesian modelling to baggr users who have no background in either, so this might be a good start!

be-green · 2020-01-26T15:15:00Z

No rush, just figured it might be useful!

wwiecek

Need to delete .Rd's and check for merge errors. Vignette should open a bit differently. Otherwise it's great

wwiecek · 2020-02-07T09:51:03Z

vignettes/baggr_model_convergence.Rmd

+bibliography: baggr.bib
+---
+
+The `baggr` package makes running Bayesian meta-analysis models very easy, but that ease can also obscure some things that go on behind the scenes. Especially with low-data settings, say with only a few measurements, you may get cryptic error messages when you run your models, like


Just baggr.

Please explain in the very first sentence what this vignette is about and who is it for: "This is a vignette intended for baggr users who have no or little experience with Markov Chain Monte Carlo (MCMC) programs used for Bayesian inference. It is intended to help you understand common problems that may occur when running models. Especially in when there are few data you may see cryptic error messages when you run your models, like"

Yep, that makes sense. I'll edit and re-push.

wwiecek · 2020-02-07T09:52:57Z

vignettes/baggr_model_convergence.Rmd

+Warning: There were 110 divergent transitions after warmup. Increasing adapt_delta above 0.9 may help. See
+http://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
+```
+


Here you need to explain basic terms. In few words, as if explaining to an undergraduate. What is Stan - 1 sentence. What is it for. What is MCMC. How does baggr use Stan. Why problems can occur.

By "here" I meant after "This vignette" paragraph. But before you go into any particualr issue.

I know that you will re-state this in more detail below, but sometimes people stop reading at first difficulty. So having a simple paragraph (it could be a bulletpointed mini-glossary?) will enable many of them.

That is useful feedback, I'll make these changes.

wwiecek · 2020-02-07T09:53:44Z

vignettes/baggr_model_convergence.Rmd

+
+This vignette will walk through the basic intuition for what those mean, whether to be concerned about them, and how to tune the parameters underneath the hood. There are already a number of good resources that go more in depth on the topic so first I'll point you there:
+
+[Visual MCMC Diagnostics Using the Bayesplot Package](https://mc-stan.org/bayesplot/articles/visual-mcmc-diagnostics.html)


I'd cite the extra resources after your basic explanations. Otherwise readers will click on them and go away

wwiecek · 2020-02-07T09:57:15Z

vignettes/baggr_model_convergence.Rmd

+
+# $\widehat{R}$ being too low
+
+The other warning you get, that “r hat” statistics are too high, indicates that the effective sample size (ESS) of your draws are too low and your chains have not converged. Because MCMC is not a deterministic algorithm, we run multiple chains of samples in order to check that each has converged to a good enough answer.


These are 2 mutual issues? ie correlated, but can have low R hat, low ESS, no?

Yes you are right, this is misleading at best, and at worst incorrect.

wwiecek · 2020-02-07T09:58:29Z

DESCRIPTION

@@ -29,7 +29,7 @@ Imports:
 LinkingTo: StanHeaders (>= 2.18.1), rstan (>= 2.18.1), BH (>= 1.66.0-1), Rcpp (>= 0.12.17), RcppEigen (>= 0.3.3.4.0)
 SystemRequirements: GNU make
 NeedsCompilation: yes
-RoxygenNote: 6.1.1
+RoxygenNote: 7.0.2


Minor, but please remove this and also delete .Rd files. I want to update Roxygen on my end and update .Rd files after we merge. Otherwise there will be stupid clashes when merging to devel

Alright--I found that if I wanted linking within the documentation to work I needed to update roxygen2. I can delete the note and the .rd files though.

wwiecek · 2020-02-07T09:59:17Z

man/baggr.Rd

@@ -36,7 +48,7 @@ If unspecified, the priors will be derived automatically based on data
 (and printed out in the console).}

 \item{prior_hypersd}{prior for hyper-standard deviation, used
-by Rubin and \code{"mutau"`` models; same rules apply as for }_hypermean`;}
+by Rubin and \verb{"mutau"`` models; same rules apply as for }_hypermean`;}


Is this OK? I don't get what is going on here. (My old code also looks broken)

This might have been a case where I accidentally deleted and re-added something and it got swept up in a commit unintentionally.

Please check how it renders, I think my old code also looks wrong!

wwiecek · 2020-02-07T14:40:37Z

@be-green no need to delete Roxygen Note (I just updated mine to version 7 too, that was part of the problem), but for safety good to delete .Rd files as they are constantly changing in devel. Sorry!

wwiecek · 2020-02-13T10:51:15Z

@be-green I'm ready to merge this, let me know if I can help in any way. I think you only need to push a few changes + get rid of .Rd files (or try re-making them first, I moved to new Roxygen so maybe you won't have conflicts with devel anymore? but if you still have problems, get rid of the .Rd offenders)

be-green · 2020-02-13T13:10:00Z

Ok I'll make changes, delete .rd files, and push a new commit. Sorry, I've been swamped with another project this week and haven't made as much progress on this as I'd like.

… baggr

…ocumented

wwiecek · 2020-02-20T20:14:55Z

I need to make more edits, will not incldue this for v0.4 as I'm running out of time to review.

This was referenced Jan 25, 2020

Vignette for model convergence checks + some automated (?) tests #42

Open

Document rstan::sampling() arguments #51

Open

wwiecek requested changes Feb 7, 2020

View reviewed changes

be-green added 11 commits February 14, 2020 13:43

warnings vignette

b0b5d4d

update convergence vignette, link rstan::sampling in the ... args for…

6c14579

… baggr

remove html that I accidentally included

6018a62

code formatting typo

facd403

max_treedepth warning errors

ca7de8d

reference to rstan::stan because that's where control arguments are d…

a2e605d

…ocumented

warning for high r_hats that recommends more iterations

9623065

remove file that wasn't properly cleaned up from vignette

4d495ab

silly Bayes' rule mistake on the integral

41169e6

fix weird tab where there wasn't one before

18298e1

update convergence vignette

04d18b6

be-green force-pushed the devel-convergence-vignette branch from 3d57a82 to 04d18b6 Compare February 14, 2020 19:46

Update baggr_model_convergence.Rmd

a388df2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Devel convergence vignette #53

Devel convergence vignette #53

be-green commented Jan 25, 2020

codecov-io commented Jan 25, 2020 •

edited

Loading

wwiecek commented Jan 26, 2020

be-green commented Jan 26, 2020

wwiecek left a comment

wwiecek Feb 7, 2020

be-green Feb 7, 2020

wwiecek Feb 7, 2020

wwiecek Feb 7, 2020

wwiecek Feb 7, 2020

be-green Feb 7, 2020

wwiecek Feb 7, 2020

wwiecek Feb 7, 2020

be-green Feb 7, 2020

wwiecek Feb 7, 2020

be-green Feb 7, 2020

wwiecek Feb 7, 2020

be-green Feb 7, 2020

wwiecek Feb 7, 2020

wwiecek commented Feb 7, 2020

wwiecek commented Feb 13, 2020

be-green commented Feb 13, 2020

wwiecek commented Feb 20, 2020


		This vignette will walk through the basic intuition for what those mean, whether to be concerned about them, and how to tune the parameters underneath the hood. There are already a number of good resources that go more in depth on the topic so first I'll point you there:

		[Visual MCMC Diagnostics Using the Bayesplot Package](https://mc-stan.org/bayesplot/articles/visual-mcmc-diagnostics.html)


		# $\widehat{R}$ being too low

		The other warning you get, that “r hat” statistics are too high, indicates that the effective sample size (ESS) of your draws are too low and your chains have not converged. Because MCMC is not a deterministic algorithm, we run multiple chains of samples in order to check that each has converged to a good enough answer.

Devel convergence vignette #53

Are you sure you want to change the base?

Devel convergence vignette #53

Conversation

be-green commented Jan 25, 2020

codecov-io commented Jan 25, 2020 • edited Loading

Codecov Report

wwiecek commented Jan 26, 2020

be-green commented Jan 26, 2020

wwiecek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwiecek commented Feb 7, 2020

wwiecek commented Feb 13, 2020

be-green commented Feb 13, 2020

wwiecek commented Feb 20, 2020

codecov-io commented Jan 25, 2020 •

edited

Loading