Convergence issues in PyEI_overview notebook

[matt-graham]

Submitting as part of JOSS review openjournals/joss-reviews#3397

A potential concern I have about the package is that by hiding away to some extent the details of how inference is performed, there is a danger of a user analyzing the results without being aware of the implications of potential convergence issues in the Markov chain Monte Carlo sampling. This is not a purely hypothetical scenario, as the chains sampled when fitting the example model in the pyei/intro_notebooks/PyEI_overview.ipynb notebook appear to be suffering convergence issues.

In particular the chains appear to include a non-negligible number of divergent transitions in the post-warmup stage, which is generally indicative of a problematic posterior geometry with varying scales / curvature. The R̂ convergence diagnostics estimates computed by ArviZ — 1.09 for phi_1 and 1.07 for phi_2 on a run with random_seed=20210708 — also indicate that the chains are poorly converged, as do the low effective sample size estimates.

While warning messages from PyMC3 which highlight these issues are displayed, there is no discussion in the notebook of these issues and possible remedies. Further as the target_accept parameter is already set to the very high value of 0.99, there is limited scope for further increases as recommended in the warning messages. I think it would be worthwhile to include at least some brief discussion of these issues in the notebook(s) with pointers to relevant resources in the PyMC3 or Stan documentation. Ideally there would also be an example of experimentation with approaches to mitigate these issues, for example, using additional prior knowledge to define more informative priors which might shift the posterior mass away from the problematic regions; trying alternative parametrizations - the docstring for ei_beta_binom_model_modified suggests this is already a reparametrization of the standard formulation - if the standard formulation does indeed suffer even more from convergence issues it could be worthwhile including this as an initial example in the notebook to illustrate the importance of the choice of parametrization.

[pmyteh]

I agree. I was pleased to see a brief discussion at the bottom of Introduction_to_EI.ipynb about things that can go wrong with Bayesian modelling, and that PyEI was 'more vocal' about revealing such issues to the user. Brilliant! That these issues arise and are reported by the package when running the examples provides a wonderful opportunity to highlight the relative seriousness of these issues in the context of EI work. You (the authors) will know far more about what is concerning and what is not than the majority of your userbase will.

Obviously there's something of a balance: package documentation can't replace a hefty stats textbook. But maybe a short overview of the warnings issued and pointers to interpreting them in the overview docs, together with a brief statement of why you consider the model 'good enough' to proceed despite the warnings in the example notebooks. And given that, frankly, many users will be trying these models without reading the textbook first, heading them off from the obvious footguns would be a serious form of academic service!

[matt-graham]

The changes made in ad51b5b look great to me and resolve this from my perspective, so closing.