big-picture comments

[petrelharp]

I've done a general read-through of the paper, and am recording big-picture comments here (for lack fo a better place?).

Generally: it looks great! I'd say that right now it is a general overview of the new features, with demonstrations that these new features are indeed fast. There isn't much detail on how things actually work, except in the Appendix. I think this is probably good - how much detail we want depends on the audience, and adding more detail would make it too long.

1. In the "tree sequence" section, it doesn't yet explain how mutations are put in there.
2. However, the next section does, so maybe that section should be titled "mutations" not "simulating mutations"
3. We've got nothing showing the Demography features. We might have a table of Demography ingredients? Or a diagram of a demography with a few events, with the code to produce it alongside? Right now it feels a bit abstract, and arriving at "instantaneous bottlenecks" feels like then there will be sections for many more demographic ingredients (but there is not). Alternatively, something people will wonder is "will my old demography code work" and/or "how do I translate to the new demography"; perhaps we can address that here?
4. In the ARG section, we might make a diagram showing something that's in the full ARG but not the minimal tree sequence, to make the distinction clear? As part (b) of Figure 5? ARG illustration #85
5. In Figure 7, the y-axis is on a log scale. And, maybe we should extend the x-axis to 100Mb?
6. Maybe "simulation interface" and "development model" should go first?
7. In "development model" we could give an example of how we test things, to make it clear that it's rigorous.
8. There's no discussion of when you need to do forwards simulation (selection, continuous space, etc); maybe this belongs in the Introduction?
9. Should we describe how the mutation simulation algorithm works? In an appendix? Appendix for mutation algorithm #84
10. I wonder if there's some descriptive figures we'd like to include, e.g. a conceptual one that shows simulating trees and then adding mutations (using the nice tskit svg drawing capability), or example trees simulated with and without multiple mergers. This could spice up the performance figures (which all look the same).

I'm happy to do some of these things, but thought I should make one issue instead of ten - but, maybe if others agree a thing should be done, it should be made a separate issue? I'm not sure how to deal with this.

[benjeffery]

I'm happy to do some of these things, but thought I should make one issue instead of ten - but, maybe if others agree a thing should be done, it should be made a separate issue? I'm not sure how to deal with this.

I've activated discussions on this repo - maybe it makes sense to have issues be "concrete changes or deliverables" and discussions be more open-ended?

[jeromekelleher]

Yep, good call @benjeffery, done.

[fbaumdicker]

Regarding point 9. and the size of the appendix, I am wondering whether we aim for a part in the appendix for each section, i.e. also for the gene conversion part? Or is this not necessary?

Regarding 10. I think additional figures of the resulting tree sequences for the different mechanisms, like the ones we already have in the online documentation, would really spicy up the manuscript, although I am not sure if this will violate a potential page limit.

[jeromekelleher]

I think we should add appendices explaining what we're doing, if we're doing something novel or non-obvious. For GC though, I think we're basically just implementing the WH model, right?

[fbaumdicker]

Yes, the only difference is in the internal rate computation which works differently in Hudson's ms, but the resulting distribution/model is exactly the WH model. Great, that means less work for me. :)

[jeromekelleher]

I don't think we could explain this without really getting into the guts of msprime's recombination implementation - but that was what the 2016 paper was for. Let's leave it.

[jeromekelleher]

Pulling this one out for a bit more discussion:

We've got nothing showing the Demography features. We might have a table of Demography ingredients? Or a diagram of a demography with a few events, with the code to produce it alongside? Right now it feels a bit abstract, and arriving at "instantaneous bottlenecks" feels like then there will be sections for many more demographic ingredients (but there is not). Alternatively, something people will wonder is "will my old demography code work" and/or "how do I translate to the new demography"; perhaps we can address that here?

We could have a table all right, although we're pretty heavy on display items at the moment. I've been resisting putting code chunks into the paper as I think the paper is for telling people what msprime can do, rather then how to do it (that's what the docs are for). Also, I don't want to get into talking too much about Demography modelling, as this is what the Demes paper is for.

I'm definitely open to ideas on how to improve the section though. @grahamgower, @apragsdale, any thoughts here on what might be missing from the current draft?

[grahamgower]

I made a few changes to this section in #86. But I don't have any suggestions about including a table or code --- these seem like overkill to me. I agree that the "instantaneous bottlenecks" section does stick out a bit, but not sure how to fix that.

[petrelharp]

An explanatory sentence at the start of "instantaneous bottlenecks" saying that the other demographic things were already there, but there's one new one could help it not stick out? And maybe decrease the heading size (e.g., make it a \paragraph{ } instead of a \subsection{ })?

[jeromekelleher]

Pulling this one out:

I wonder if there's some descriptive figures we'd like to include, e.g. a conceptual one that shows simulating trees and then adding mutations (using the nice tskit svg drawing capability), or example trees simulated with and without multiple mergers. This could spice up the performance figures (which all look the same).

So we are a little display-item heavy already, and I'm not sure what we'd be showing here that Figure 1 doesn't illustrate. Showing "typical" tree sequences for any of the different processes would be pretty tricky, and I'm not sure what that would achieve?

[jeromekelleher]

Hmm - I think we're writing to a different audience here. This is obvious to the people I have in mind (notably, PI X who has used ms/fastsimcoal/MaCS for umpteen years and isn't particularly keen to change). Who are you thinking about?

[petrelharp]

Well, I guess I had in mind the graduate students who are getting in to simulation? In some ways that audience is better because (a) they'll actually read the text of the paper, and (b) they don't have a big library of scripts already. But, I think we can address both - this doesn't get in the way of the already-experts?

And, the figures help make the point that there's more neat stuff we can do (eg visualization) when we have the trees? If you like I can draft up the combined figures?

[jeromekelleher]

Yeah, OK. That does put a different spin on things.

I do think it's important to get through to the old-timers though, as they will (I assume) be influential in the path that graduate students take when getting in to simulations. My fear is that if we write what looks like "an introduction to coalescent simulation" it'll be ignored by this audience. We already have plenty introductory material out there on how to run simulations (online docs), and what they're for (msprime book chapter).

[petrelharp]

er, so what's your conclusion about the other examples? (GC tree? sweep tree?)

[jeromekelleher]

I don't know to be honest - how about we write the paper without these visualisations first and then see if adding them in would improve it?

[jeromekelleher]

@petrelharp, a quick response on this one:

And, for the other figures we could stick in a picture of the trees next to the performance plot, that depicts the process in question. For instance:

• next to the GC figure we could add a short tree sequence with a small GC-induced tree in the middle

We already have that in the docs here, but it subsequently caused confusion (https://tskit-dev.slack.com/archives/CVC0GB5C1/p1626798867096800) - I think we'll spend a lot of time messing around trying to get nice looking trees, trying to explain to people what they are looking at, and they'll just end up confusing people anyway.

[petrelharp]

Hm - not sure I agree. The slack conversion you linked to said the confusion was from the words there, not from the figure? Arguably, the figure should have helped, because it showed the right thing happening? But, I'm happy to skip these illustrations.

[jeromekelleher]

That's true. I guess it comes back to an earlier point - I don't really want to get into explaining what gene conversion (or any of the other processes) is. That's a different paper IMO.

[petrelharp]

Good point; I agree with that.