[REVIEW]: sbi reloaded: a toolkit for simulation-based inference workflows

[editorialbot]

Submitting author: @janfb (Jan Boelts)
Repository: https://github.com/sbi-dev/sbi
Branch with paper.md (empty if default branch): joss-submission-2024
Version: v0.23.2
Editor: @boisgera
Reviewers: @arnauqb, @francois-rozet
Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/2ce76f5723a6dfe80cd30fd445c4d2ce"><img src="https://joss.theoj.org/papers/2ce76f5723a6dfe80cd30fd445c4d2ce/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/2ce76f5723a6dfe80cd30fd445c4d2ce/status.svg)](https://joss.theoj.org/papers/2ce76f5723a6dfe80cd30fd445c4d2ce)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@arnauqb & @francois-rozet, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @boisgera know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @arnauqb

📝 Checklist for @francois-rozet

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.98  T=0.32 s (766.3 files/s, 182497.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                         167           6632          10783          24368
Jupyter Notebook                22              0           8641           2344
Markdown                        31            575              0           1889
TeX                              1            126              0            989
YAML                            11             77             13            515
Text                             1             33              0            169
TOML                             1             12              8            126
JSON                             2              3              0             93
SVG                              2              0              1             86
JavaScript                       2             10              0             57
CSS                              1             11              7             51
Dockerfile                       1              3              3              9
-------------------------------------------------------------------------------
SUM:                           242           7482          19456          30696
-------------------------------------------------------------------------------

Commit count by author:

   303	janfb
   182	Michael Deistler
   125	Alvaro Tejero-Cantero
    92	Jan
    72	Jan Boelts
    68	michaeldeistler
    66	michael
    65	jan-matthis
    56	michael.deistler
    28	Jan-Matthis
    21	meteore
    18	manuelgloeckler
    17	Fabio Muratore
    15	Guy Moss
    12	Peter Steinbach
     9	pedro goncalves
     7	tomMoral
     6	Sebastian Bischoff
     6	coschroeder
     6	jnsbck
     5	Julia Linhart
     5	LouisRouillard
     5	zinaStef
     4	Benjamin Kurt Miller
     4	Thomas Moreau
     4	jsvetter
     3	Imahn
     3	Matthijs Pals
     3	Yves Bernaerts
     3	augustes
     3	danielmk
     3	dgreenberg
     2	Conor Durkan
     2	Gilles Louppe
     2	Maternus
     2	Pedro L. C. Rodrigues
     2	Pedro Rodrigues
     2	milagorecki
     2	rdgao
     1	A. Ziaeemehr
     1	Abdul Samad
     1	Abdul Samad Siddiqui
     1	Abolfazl Ziaeemehr
     1	Alexandre Gramfort
     1	Ankush Checkervarty
     1	Arnaud Delaunoy
     1	Cornelius
     1	David Greenberg
     1	Eslam Khaled
     1	Felix Pei
     1	Harry Fu Yu
     1	Imahn Shekhzadeh
     1	JBeckUniTb
     1	JH Macke
     1	Janne Lappalainen
     1	Kristof Schröder
     1	Miles Cranmer
     1	Narendra Mukherjee
     1	Nastya Krouglova
     1	Nathan Musoke
     1	Nick Tolley
     1	Pietro Monticone
     1	Pizza GitHub
     1	Richard Gao
     1	Seth Axen
     1	Thomas Gessey-Jones
     1	Victor Buendía
     1	Yoav Ram
     1	bkmi
     1	felixp8
     1	max-dax
     1	tbmiller-astro
     1	theo
     1	theogruner
     1	Álvaro Tejero-Cantero

[editorialbot]

Paper file info:

🚨 Wordcount for paper.md is 2241

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: Apache License 2.0 (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1093/biomet/asp052 is OK
- 10.1201/9781315117195 is OK
- 10.1098/rsif.2008.0172 is OK
- 10.12751/nncn.bc2018.0222 is OK
- 10.1186/s12868-017-0370-3 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: ABCpy
- No DOI given, and none found for title: pydelfi: Density Estimation Likelihood-Free Infere...
- No DOI given, and none found for title: nflows:  Normalizing flows in PyTorch
- No DOI given, and none found for title: Zuko - Normalizing flows in PyTorch
- No DOI given, and none found for title: Pyro: Deep Universal Probabilistic Programming
- No DOI given, and none found for title: DELFI: Density Estimation Likelihood-Free Inferenc...
- No DOI given, and none found for title: joblib
- No DOI given, and none found for title: Neural spline flows
- No DOI given, and none found for title: ELFI: engine for likelihood-free inference
- No DOI given, and none found for title: PyTorch: An Imperative Style, High-Performance Dee...
- No DOI given, and none found for title: Sequential monte carlo without likelihoods
- No DOI given, and none found for title: Python 3 Reference Manual
- No DOI given, and none found for title: Flexible statistical inference for mechanistic mod...
- No DOI given, and none found for title: Neural Approximate Sufficient Statistics for Impli...
- No DOI given, and none found for title: Fast \varepsilon-free inference of simulation mode...
- No DOI given, and none found for title: Normalizing flows for probabilistic modeling and i...
- No DOI given, and none found for title: Sequential neural likelihood: Fast likelihood-free...
- No DOI given, and none found for title: Masked autoregressive flow for density estimation
- No DOI given, and none found for title: Neural spline flows
- No DOI given, and none found for title: Automatic posterior transformation for likelihood-...
- No DOI given, and none found for title: Benchmarking simulation-based inference
- No DOI given, and none found for title: Likelihood-free inference with emulator networks
- No DOI given, and none found for title: Truncated marginal neural ratio estimation
- No DOI given, and none found for title: Variational methods for simulation-based inference
- No DOI given, and none found for title: Training deep neural density estimators to identif...
- No DOI given, and none found for title: Efficient bayesian experimental design for implici...
- No DOI given, and none found for title: Bayesian inference for biophysical neuron models e...
- No DOI given, and none found for title: The frontier of simulation-based inference
- No DOI given, and none found for title: Variational inference with normalizing flows
- No DOI given, and none found for title: Inference Compilation and Universal Probabilistic ...
- No DOI given, and none found for title: Likelihood-free mcmc with amortized approximate ra...
- No DOI given, and none found for title: Sequential neural methods for likelihood-free infe...
- No DOI given, and none found for title: Amortised inference for mechanistic models of neur...
- No DOI given, and none found for title: Adam: A Method for Stochastic Optimization
- No DOI given, and none found for title: Bayesian optimization for likelihood-free inferenc...
- No DOI given, and none found for title: Mixture density networks
- No DOI given, and none found for title: Sequential monte carlo without likelihoods
- No DOI given, and none found for title: Adaptive approximate Bayesian computation
- No DOI given, and none found for title: On contrastive learning for likelihood-free infere...
- No DOI given, and none found for title: Group equivariant neural posterior estimation
- No DOI given, and none found for title: A Crisis In Simulation-Based Inference? Beware, Yo...
- No DOI given, and none found for title: Validating Bayesian inference algorithms with simu...
- No DOI given, and none found for title: Flexible and efficient simulation-based inference ...
- No DOI given, and none found for title: Auto-Encoding Variational Bayes
- No DOI given, and none found for title: Robust Neural Posterior Estimation and Statistical...
- No DOI given, and none found for title: Differentiable likelihoods for fast inversion of’l...
- No DOI given, and none found for title: Truncated proposals for scalable and hassle-free s...
- No DOI given, and none found for title: Optimal simulation-based Bayesian decisions
- No DOI given, and none found for title: Contrastive neural ratio estimation
- No DOI given, and none found for title: Compositional score modeling for simulation-based ...
- No DOI given, and none found for title: Jana: Jointly amortized neural approximation of co...
- No DOI given, and none found for title: Simulation-based Inference with the Generalized Ku...
- No DOI given, and none found for title: Sensitivity-Aware Amortized Bayesian Inference
- No DOI given, and none found for title: Neural Score Estimation: Likelihood-Free Inference...
- No DOI given, and none found for title: Inference compilation and universal probabilistic ...
- No DOI given, and none found for title: MADE: Masked Autoencoder for Distribution Estimati...
- No DOI given, and none found for title: Flow Matching for Scalable Simulation-Based Infere...
- No DOI given, and none found for title: Generalized Bayesian inference for scientific simu...
- No DOI given, and none found for title: Amortized Bayesian Decision Making for simulation-...
- No DOI given, and none found for title: A crisis in simulation-based inference? beware, yo...
- No DOI given, and none found for title: Embryo-uterine interaction coordinates mouse embry...
- No DOI given, and none found for title: Meta-learning families of plasticity rules in recu...
- No DOI given, and none found for title: Neural networks enable efficient and accurate simu...
- No DOI given, and none found for title: Variational methods for simulation-based inference
- No DOI given, and none found for title: Neural simulation-based inference approach for cha...
- No DOI given, and none found for title: Black-box Bayesian inference for economic agent-ba...
- No DOI given, and none found for title: Investigating the impact of model misspecification...
- No DOI given, and none found for title: Neural posterior domain randomization
- No DOI given, and none found for title: Skewed distribution of spines is independent of pr...
- No DOI given, and none found for title: Adversarial robustness of amortized Bayesian infer...
- No DOI given, and none found for title: Calibrating Agent-based Models to Microdata with G...
- No DOI given, and none found for title: Sequential neural posterior and likelihood approxi...
- No DOI given, and none found for title: Bayesian model comparison for simulation-based inf...
- No DOI given, and none found for title: Simulation-based inference using surjective sequen...
- No DOI given, and none found for title: L-c2st: Local diagnostics for posterior approximat...
- No DOI given, and none found for title: Sampling-based accuracy testing of posterior estim...
- No DOI given, and none found for title: Score-Based Generative Modeling through Stochastic...
- No DOI given, and none found for title: All-in-one simulation-based inference
- No DOI given, and none found for title: Flow Matching for Generative Modeling
- No DOI given, and none found for title: sbijax: Simulation-based inference in JAX
- No DOI given, and none found for title: LAMPE: Likelihood-free AMortized Posterior Estimat...
- No DOI given, and none found for title: Swyft: A system for scientific simulation-based in...
- No DOI given, and none found for title: Simulation-based inference with the Python Package...
- No DOI given, and none found for title: Combined statistical-mechanistic modeling links io...
- No DOI given, and none found for title: Generalized Bayesian inference for scientific simu...

❌ MISSING DOIs

- 10.7717/peerj-cs.1516 may be a valid DOI for title: PyMC: a modern, and comprehensive probabilistic pr...
- 10.1038/nature09319 may be a valid DOI for title: Statistical inference for noisy nonlinear ecologic...
- 10.1093/genetics/162.4.2025 may be a valid DOI for title: Approximate Bayesian computation in population gen...
- 10.21105/joss.02505 may be a valid DOI for title: sbi: A toolkit for simulation-based inference
- 10.1080/01621459.2013.864178 may be a valid DOI for title: Expectation propagation for likelihood-free infere...
- 10.1101/669218 may be a valid DOI for title: Approximate Bayesian Inference for a Mechanistic M...
- 10.5705/ss.202015.0340 may be a valid DOI for title: Learning summary statistic for approximate Bayesia...
- 10.1093/genetics/162.4.2025 may be a valid DOI for title: Approximate Bayesian computation in population gen...
- 10.1007/s11222-009-9116-0 may be a valid DOI for title: Non-linear regression models for Approximate Bayes...
- 10.1214/20-ba1238 may be a valid DOI for title: Likelihood-free inference by ratio estimation
- 10.1214/18-ba1121 may be a valid DOI for title: Efficient acquisition rules for model-based approx...
- 10.1073/pnas.2207632119 may be a valid DOI for title: Energy-efficient network activity from disparate c...
- 10.1109/hpec49654.2021.9622796 may be a valid DOI for title: A comparison of automatic differentiation and cont...
- 10.1109/tnnls.2020.3042395 may be a valid DOI for title: BayesFlow: Learning complex stochastic models with...
- 10.1109/cvpr.2016.90 may be a valid DOI for title: Deep residual learning for image recognition
- 10.1101/2024.08.21.608979 may be a valid DOI for title: Differentiable simulation enables large-scale trai...
- 10.1002/9781119585640.ch2 may be a valid DOI for title: Deep learning in population genetics
- 10.1038/s41586-022-04428-3 may be a valid DOI for title: A biophysical account of multiplication by a singl...
- 10.1016/j.neuron.2023.11.006 may be a valid DOI for title: Disinhibition by VIP interneurons is orthogonal to...
- 10.1016/j.celrep.2023.112906 may be a valid DOI for title: Theta and gamma rhythmic coding through two spike ...
- 10.3847/1538-4357/ac7b84 may be a valid DOI for title: Accelerated Bayesian SED modeling using amortized ...
- 10.1103/physrevd.109.083536 may be a valid DOI for title: Field-level simulation-based inference of galaxy c...
- 10.1016/j.neuroimage.2023.120278 may be a valid DOI for title: Bayesian inference of a spectral graph model for b...
- 10.1101/2023.01.31.526269 may be a valid DOI for title: Simulation-based inference for efficient identific...
- 10.1093/bioinformatics/bty361 may be a valid DOI for title: pyABC: distributed, likelihood-free inference
- 10.21105/joss.01143 may be a valid DOI for title: ArviZ a unified library for exploratory analysis o...
- 10.21105/joss.05702 may be a valid DOI for title: BayesFlow: Amortized Bayesian workflows with neura...
- 10.1101/2024.08.21.608969 may be a valid DOI for title: Deep inverse modeling reveals dynamic-dependent in...
- 10.2139/ssrn.4982890 may be a valid DOI for title: A Comprehensive Guide to Simulation-based Inferenc...
- 10.1016/j.bpj.2022.11.920 may be a valid DOI for title: Simulation-based inference of single-molecule forc...
- 10.1016/j.neunet.2023.03.040 may be a valid DOI for title: Amortized Bayesian inference on generative dynamic...

❌ INVALID DOIs

- None

[arnauqb]

Review checklist for @arnauqb

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/sbi-dev/sbi?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@janfb) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[francois-rozet]

Review checklist for @francois-rozet

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/sbi-dev/sbi?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@janfb) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[francois-rozet]

@boisgera I have read the paper and filled the checklist. The improvements are significant and welcome. I only have two concerns:

1. I could not find in the documentation how to use normalizing flows implemented in Zuko (Rozet, 2023). This feature is mentioned in the paper.

2. In the "Related software" section, the reference to LAMPE should cite Rozet, Delaunoy and Miller (2021).

   The Probabilists (2024) package offers neural posterior and neural ratio estimation, primarily targeting SBI researchers with a low-level API and full flexibility over the training loop (Lampe stopped being maintained in July 2024).

   I would suggest the following sentence:

   The lampe (Rozet, 2021) package offers neural posterior and neural ratio estimation, primarily targeting SBI researchers with a low-level API and full flexibility over the training loop. Its development has stopped in favor of the sbi project in July 2024.

In addition, I would suggest not to provide the source code directly in the documentation, but to link to the code repository instead. This is possible with Sphinx, I don't know if it is with mkdocs.

[arnauqb]

I'm happy with how the repo is structured and I think all the tests and documentation are of very high standard. I only have a few suggestions:

1. I would add a short paragraph discussing how sbi fits into the probabilistic programming space. Many people may be aware of Pyro, Numpyro, Turing.jl, etc. and may wonder what is the difference between sbi and these packages and how it complements them. This would also help readers to better understand if sbi is the right tool to use for their needs.

2. Shameless plug (I am one of the authors), but I will suggest it since the joss editorial bot also highlighted it. Consider adding https://joss.theoj.org/papers/10.21105/joss.05776 to the related work. Obviously only if you see it fit!

3. I tried running the tests locally and even with pytest -n auto -m "not slow and not gpu" the tests took a considerable amount of time. I imagine this may not be great when one wants to use the unittests as a quick way to check things didn't break during development.

[boisgera]

@arnauqb, @francois-rozet

Thanks a lot for these reviews, many great points there!

As @arnauqb mentions, several DOIs are missing from the bibliography. The bibliography should be updated to include DOIs whenever they exist (this is will be checked in post-review by myself and the editor in chief in any case).

Let's allow @janfb to consider this feedback and get back to us.

[janfb]

Thanks a lot @francois-rozet and @arnauqb for your timely and constructive reviews!

Thanks @boisgera for the update.

We will work on the mentioned points and get back to you as as soon as possible.

[janfb]

@francois-rozet thanks again for your review!

1. I could not find in the documentation how to use normalizing flows implemented in Zuko (Rozet, 2023). This feature is mentioned in the paper.

Good catch! We update the documentation accordingly, see here: sbi-dev/sbi#1387

3. In the "Related software" section, the reference to LAMPE should cite Rozet, Delaunoy and Miller (2021).> > The Probabilists (2024) package offers neural posterior and neural ratio estimation, primarily targeting SBI researchers with a low-level API and full flexibility over the training loop (Lampe stopped being maintained in July 2024).

   I would suggest the following sentence:

   The lampe (Rozet, 2021) package offers neural posterior and neural ratio estimation, primarily targeting SBI researchers with a low-level API and full flexibility over the training loop. Its development has stopped in favor of the sbi project in July 2024.

Good point, thanks. We updated the paper and references accordingly.

In addition, I would suggest not to provide the source code directly in the documentation, but to link to the code repository instead. This is possible with Sphinx, I don't know if it is with mkdocs.

Thanks for your feedback! Just to clarify—are you referring to the tutorial notebooks on the website? We use MkDocs and nbconvert to display the latest version of the notebooks for each release (as selected on the site). Plus, each notebook includes links to the corresponding file in the GitHub repository. Does that address your concern about the source code in the documentation?

[janfb]

I'm happy with how the repo is structured and I think all the tests and documentation are of very high standard. I only have a few suggestions:

@arnauqb thanks again for your review!

1. I would add a short paragraph discussing how sbi fits into the probabilistic programming space. Many people may be aware of Pyro, Numpyro, Turing.jl, etc. and may wonder what is the difference between sbi and these packages and how it complements them. This would also help readers to better understand if sbi is the right tool to use for their needs.

Very good point! We added a paragraph at the beginning of the related software section:

"Simulation-based inference methods implemented in the sbi package require only access
to simulated data, which can also be generated offline in other programming languages or
frameworks. This sets sbi apart from toolboxes for traditional Bayesian inference,
such as MCMC-based methods [@abril2023pymc; @bingham2019pyro; @gelman2015stan], which
rely on likelihood evaluations, and from probabilistic programming languages (e.g., Pyro
[@bingham2019pyro], NumPyro [@phan2019composable], Stan [@gelman2015stan], or Turing.jl
[@ge2018t]), which typically require the simulator to be differentiable and implemented
within their respective frameworks [@quera-bofarull2023]."

3. Shameless plug (I am one of the authors), but I will suggest it since the joss editorial bot also highlighted it. Consider adding https://joss.theoj.org/papers/10.21105/joss.05776 to the related work. Obviously only if you see it fit!

It makes sense to add this paper as well, thanks! We added it to the new paragraph (see above).

5. I tried running the tests locally and even with pytest -n auto -m "not slow and not gpu" the tests took a considerable amount of time. I imagine this may not be great when one wants to use the unittests as a quick way to check things didn't break during development.

Another good point, thanks! We are currently discussion how to make the tests faster, e.g., by improving the test suite itself, and by reducing the set of core tests used during CI, (see sbi-dev/sbi#1378).

Does this address all the comments and concerns you had?

[janfb]

@boisgera I pushed the edits in the main text and the references (fixed DOIs) to the submission branch (https://github.com/sbi-dev/sbi/tree/joss-submission-2024).

We have question regarding the formatting of the author list: We have several groups of authors who contributed equally at different levels. E.g., Michael Deistler and me are both maintainers and equally contributing first authors. The following four authors on the list are "core contributors" and the next five are "major contributors", setting them apart from regular "contributors". Is there a way to format the groups with superscripts in the JOSS markdown template (similar to how we did it on arvix, https://arxiv.org/pdf/2411.17337)?

Thanks!

[francois-rozet]

Thank you for addressing my comments. Everything seems good to me. Great work!

Just to clarify—are you referring to the tutorial notebooks on the website?

I am referring to the source code present in the API section of the website. These code blocks are not very legible without code highlighting, and the code lines are not aligned with the line numbers on the left.

Another issue with these source code blocks is that they don't allow to search the code base (where is this function defined, what does it do, what are the arguments, ...). Linking to the source code in the repo is much more valuable.

For example, in the Zuko library, the code </> button

links to https://github.com/probabilists/zuko/blob/469a68e2dd161c2c7e01745549f56baca7c503bb/zuko/flows/autoregressive.py#L213-L308

I don't know if this is possible in mkdocs, but if it is, you should consider it. The tutorials are very good.

[janfb]

Thanks for clarifying @francois-rozet! Agreed. I created an issue and we will work on this during the upcoming hackathon where a major re-work of the documentation is planned.

[arnauqb]

Thank you for addressing my comments @janfb . Good to go from my part!

[boisgera]

@boisgera I pushed the edits in the main text and the references (fixed DOIs) to the submission branch (https://github.com/sbi-dev/sbi/tree/joss-submission-2024).

Great, thank you!

We have question regarding the formatting of the author list: We have several groups of authors who contributed equally at different levels. E.g., Michael Deistler and me are both maintainers and equally contributing first authors. The following four authors on the list are "core contributors" and the next five are "major contributors", setting them apart from regular "contributors". Is there a way to format the groups with superscripts in the JOSS markdown template (similar to how we did it on arvix, https://arxiv.org/pdf/2411.17337)?

Ah, that's a good question. I have never edited a paper with this need before, let me ask my fellow co-editors and get back to you!

[boisgera]

@janfb Wrt to the superscript for authors issue:

I have not received any answers (yet) from my co-editors. Therefore I suspect that it's not something that's been asked before. Let me have a look at the document processor (https://github.com/openjournals/inara) and I'll get back to you.

[boisgera]

@janfb Wrt to the superscript for authors issue:

I have not received any answers (yet) from my co-editors. Therefore I suspect that it's not something that's been asked before. Let me have a look at the document processor (https://github.com/openjournals/inara) and I'll get back to you.

To: @janfb

AFAICT there is a bug in inara when you try to add a custom footnote for a corresponding author; see openjournals/inara#105. I advise you to follow this issue. If no solution in a reasonable time frame, then we'll reconsider your options. Is that a plan that works for you?

[janfb]

Hi @boisgera thanks a lot for looking into this. Yes, I will have a look at the issue for editing the author list and let you know.

Thanks!