recapitation attempts to use null genomes

[Hongjinwu]

edit: Scroll down for the underlying issue.

the code:
import pyslim
import tskit
import msprime

orig_ts = tskit.load("test_true.trees")

demography = msprime.Demography()
demography.add_population(name="pop_0", initial_size=10000)
demography.add_population(name="p1", initial_size=10000)
demography.add_population(name="p2", initial_size=10000)
demography.add_population(name="p3", initial_size=10000)
demography.add_population(name="p4", initial_size=10000)

rts = pyslim.recapitate(orig_ts, ancestral_Ne=1e4)
rts.dump("true_recapitated.trees")

the issue:
Traceback (most recent call last):
File "/sim_data/sim28/recapitate.py", line 14, in
rts = pyslim.recapitate(orig_ts, ancestral_Ne=1e4)
File "/home/miniconda3/lib/python3.9/site-packages/pyslim/methods.py", line 77, in recapitate
raise ValueError(message)
ValueError: Not all roots of the provided tree sequence are at the time expected by recapitate(). This could happen if you've simplified in python before recapitating (fix: don't simplify first). If could also happen in other situations, e.g., you added new individuals without parents in SLiM during the course of the simulation with sim.addSubPop(), in which case you will probably need to recapitate with msprime.sim_ancestry(initial_state=ts, ...). (Expected root time: 27372; Observed root times: [0.0, 27372.0])

and I do not understand why the there is a population named "pop_0" in the true trees. but "pop_0" dose not exit in my slim simulation code.
1 first(){
sim.addSubpop("p1",50,haploid=T); //add population with size 50
sim.addSubpop("p2",50,haploid=T);
sim.addSubpop("p3",50,haploid=T);
sim.addSubpop("p4",50,haploid=T);
}
dose anyone know how to fix this?

[bhaller]

Hi! I think pop_0 is just a placeholder occupying slot 0 in the populations table. According to the present design, p1 needs to be in slot 1, p2 in slot 2, etc., which leaves slot 0 empty, so we put a placeholder there. pop_0 should contain no individuals/nodes, and can simply be ignored. You are including it in your demography, however, with an initial size of 10,000; just don't do that? If that does not solve the problem, then please provide the complete SLiM code for your simulation (and please make it a minimal script that still reproduces the problem, stripped of all unnecessary cruft). @petrelharp might have more to say on this. (Peter: I tried to find mention of this problem in the pyslim doc, and did not succeed; perhaps it needs to be made more prominent?)

[Hongjinwu]

Thanks for the answer. The defined demography in the python script is not used in recapitate function (I forgot to comment them). Indeed the defined demography is for sim_ancestor() function and it will give another error like "infinite time for coalescent ......., balabala"
Another thing I need to say is that I didn't simplify the tree sequence before trying to recapitate

[bhaller]

OK. Please provide a complete, minimal SLiM script that can be used to reproduce this problem, and please also provide a correct and minimal Python script to reproduce the problem.

[Hongjinwu]

Ok thank you. I don't have the scripts at hand and will post them tomorrow. Thank you again 获取Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Ben Haller ***@***.***> Sent: Tuesday, December 24, 2024 10:15:24 PM To: tskit-dev/pyslim ***@***.***> Cc: Hongjin Wu ***@***.***>; Author ***@***.***> Subject: Re: [tskit-dev/pyslim] pyslim.recapitate error (Issue #357) OK. Please provide a complete, minimal SLiM script that can be used to reproduce this problem, and please also provide a correct and minimal Python script to reproduce the problem. ― Reply to this email directly, view it on GitHub<#357 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO4Y6BD6VT5G4HOH3TRFYM32HFT7ZAVCNFSM6AAAAABUECBPN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRRGE3TGNRWGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[Hongjinwu]

recapitate.py.zip
sweep_sim.slim.zip

please find the scripts in the attachments. Thanks.
And if I use msprime.sim_ancestry() to recaptate, it will give an error:
"msprime._msprime.LibraryError: Infinite waiting time until next simulation event."

[bhaller]

Thanks. It is Christmas here; I imagine @petrelharp will have a look at this some time after that. :->

[Hongjinwu]

Ok thanks. Merry Christmas. 获取Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Ben Haller ***@***.***> Sent: Wednesday, December 25, 2024 10:10:52 PM To: tskit-dev/pyslim ***@***.***> Cc: Hongjin Wu ***@***.***>; Author ***@***.***> Subject: Re: [tskit-dev/pyslim] pyslim.recapitate error (Issue #357) Thanks. It is Christmas here; I imagine @petrelharp<https://github.com/petrelharp> will have a look at this some time after that. :-> ― Reply to this email directly, view it on GitHub<#357 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO4Y6BCMQALZY2FGXBB3TN32HK4GZAVCNFSM6AAAAABUECBPN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRRHEYDKNZZHA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[petrelharp]

Thanks for the report, @Hongjinwu - it turns out that recapitate is not dealing properly with haploids. This is a bug (I'm not yet sure how bad).

Here is a MWE (substantially pared down from your code):

initialize()
{
	initializeSLiMModelType("nonWF"); // non Wright Fisher model: useful to model haploids and HGT
	defineConstant("L", 1e5); // chromosome length
	defineConstant("Ne", 100);
	
	initializeTreeSeq();
	initializeMutationRate(6e-7);
	initializeRecombinationRate(0);
	initializeMutationType("m1", 1, "f", 0.0);
	initializeGenomicElementType("g1", m1, 1.0);
	initializeGenomicElement(g1, 0, L-1);
}


reproduction(NULL)
{
	nbOffspring = rpois(1, individual.tagF);
	for (i in seqLen(nbOffspring)) {
          subpop.addRecombinant(individual.genome1, NULL, NULL, NULL, NULL, NULL);
          // THIS WORKS:
          // subpop.addRecombinant(individual.genome1, NULL, NULL, individual.genome1, NULL, NULL);
    }
}

1 first(){
	sim.addSubpop("p1",5,haploid=T);
    inds = p1.individuals;
    inds.tagF = rep(1.0, p1.individualCount);
}
early(){
 	sim.recalculateFitness();

    inds = p1.individuals;
    inds[inds.age > 0].fitnessScaling = 0.0;
    inds = inds[inds.age == 0];
    fit = p1.cachedFitness(inds.index);

    inds.fitnessScaling = 1 / min(fit);
    inds.tagF = (Ne / size(inds)) * (1 / mean(fit)) * fit;
}

27 late() {
			sim.treeSeqOutput("test_true.trees");
			sim.simulationFinished();
}

and then

import pyslim
import tskit
import msprime

orig_ts = tskit.load("test_true.trees")

t = orig_ts.first()

root_times = set([orig_ts.node(r).time for r in t.roots])
print(f"root times: {root_times}")

non_null_root_times = set([orig_ts.node(r).time for r in t.roots if not orig_ts.node(r).metadata['is_null']])

print(f"non-null root times: {non_null_root_times}")
assert len(non_null_root_times) == 1

I'll follow up with a workaround for you.

[petrelharp]

Okay - here's a workaround:

# remove null genomes
samples = [n for n in orig_ts.samples() if not orig_ts.node(n).metadata['is_null']]
simp_ts = orig_ts.simplify(samples, keep_input_roots=True)

rts = pyslim.recapitate(simp_ts, ancestral_Ne=1e1)

(Note I'm setting a small Ne just so it runs fast.)

[bhaller]

Interesting. Note that the next SLiM will not use null genomes in a pure haploid model; but will still use them for the haploids in, say, a model of haplodiploidy. So both way of recording haploids will be possible, depending on the model. Just injecting that into the conversation here, since it might influence the fix you decide to do.

[petrelharp]

Well, thankfully we have that sanity check in; otherwise this would be quite a bad and silent error!

[Hongjinwu]

Thanks very much for the help!