Need method to generate VCF aligned with reference sequence in DNA damage simulations

[juleslr13]

I'm working on research project of DNA damage simulation pipeline for Ancestry Inference where I:

1. Generate tree sequences with msprime
2. Create FASTA files from these sequences
3. Introduce damage to create FASTQ files for analysis (via [https://github.com/Jazpy/adnator/] )
4. Need corresponding VCF files with exact positions for comparison

Current issue

I notice that write_vcf() doesn't take a reference sequence parameter, while write_fasta() does accept reference_sequence. This creates a misalignment between my files:

FASTA generation allows reference sequence
ts.write_fasta("output.fa", reference_sequence=ref_seq)

VCF generation has no equivalent parameter
ts.write_vcf("output.vcf") # No reference_sequence parameter available

While write_vcf() has a position_transform parameter, this only allows approximate position adjustments, not ensuring that alleles are correctly aligned to a reference.

Position inconsistency issue

Even when using the same tree sequence for both outputs, the positions in the VCF are not consistent with the FASTA. The VCF positions from ts.write_vcf() start at 0 by default, and don't correspond to the actual positions in the reference sequence. This creates fundamental problems for my simulation pipeline, as I need the VCF to reflect the exact same positions as in the FASTA file to properly analyze the effects of DNA damage against a known reference.

Based on my observations, write_vcf() appears to use relative positions where mutations are encoded as discrete markers beginning at 0. For FASTA generation, this mask is mapped to physical genomic coordinates by replacing unmasked positions (0s) with reference nucleotides. However, there is no direct mechanism to export VCF positions as absolute genomic coordinates rather than normalized values, creating a mismatch between the VCF’s relative positions and the FASTA’s physical positions.

Impact on DNA damage analysis

In ancient DNA/damaged sample pipelines precise VCF-FASTA alignment is critical because:

1. Damage artifacts: Misaligned positions make it impossible to distinguish true variants from damage-induced errors (e.g., C→T deamination)
2. Coverage gaps: Damage creates non-random missing data that must be precisely mapped to validate sequencing depth patterns
3. Benchmarking: Our damage correction methods require 1:1 positional correspondence to quantify false positives/negatives
4. Reference scaling: With chromosome-scale references (GRCh38), even minor positional mismatches invalidate downstream analyses

Current workarounds using relative positions (0-1 range) or approximate alignments fail to meet these requirements for real-world damaged data analysis.

Attempted workarounds

I've tried several approaches without success:

1. Applying position offsets through position_transform, but this proved too approximate and failed to achieve precise positioning.
2. Using FASTA alignment tools like minimap2 to align sequences and derive position mappings, but this is computationally expensive and still produces imprecise alignments.

These workarounds produce approximations when precision is critical. In my pipeline, the FASTA files are the "ground truth" and serve as the baseline for all comparisons. The VCF needs to perfectly match this ground truth, not approximate it. The current methods introduce unnecessary complexity and error into what should be a straightforward alignment.

Desired functionality

I need to generate VCF files that precisely match the genomic positions and mutations in my FASTA files before damage is introduced. This alignment is critical for realistic comparisons between original and damaged sequences.

My ideal solution would be either:

1. A reference_sequence parameter for write_vcf()
2. A method to ensure VCF sites align perfectly with FASTA positions

Is there a recommended approach to ensure exact position/allele alignment between FASTA and VCF outputs? Or should this functionality be added?

Perhaps write_vcf() could be enhanced to accept a reference_sequence parameter (similar to write_fasta()), since write_fasta() already successfully applies mutations at precise positions on the reference sequence - implementing this for VCF output would ensure perfect alignment between VCF and FASTA files and eliminate the need for approximate position transformations.

Would this be feasible? I'm happy to help test implementations
Thank you very much for your help.

PS : While discussion #2838 addresses 0/1-based coordinate conflicts, this request focuses on biological alignment - ensuring VCF positions/REF alleles exactly match reference-anchored FASTA outputs, crucial for damage artifact analysis. The solutions proposed there don't resolve reference sequence mismatches.

[hyanwong]

How are you adding mutations to your tree sequence? If you are using msprime.sim_mutations() then these should all be at integer positions, and it should be easy to use the position_transform parameter to simply add 1, I think?

[juleslr13]

Thank you for suggesting the RateMap approach, I think this directly addresses my issue!

To implement this correctly with chr21 (GRCh38), I need to clarify:

When creating the RateMap to skip NNNN regions, what's the best way to determine the exact positions for simulation? For example, chr21 has:

• Initial NNNN blocks (from 5,000,000 to ~5,010,000)
• Intermittent coding regions (e.g., "...NNNACGTNNN...")
• Multiple subsequent blocks of masked/unmasked regions

Should I:

1. Start positions at 0 and map NNNN→NaN, coding→rate?
2. Start at the first non-N base and use absolute reference coordinates?
3. Parse the entire reference to build a complete map of all masked/unmasked regions?

I want to ensure VCF positions match real genomic coordinates when I eventually call write_vcf() and write_fasta().

Thanks again for your help!

[hyanwong]

Note that you can also simulate all the regions (including those that are N), then subsequently mark some parts as missing using delete_intervals or keep_intervals. Although you might waste some extra CPU doing this, it should retain node identities across missing segments, and might be the easiest way to deal with intermittent known regions.

[hyanwong]

(and I would just use the "standard" absolute reference, and e.g. mark all of 0...5,010,000 as missing. It's confusing to have to remap positions all the time). It's up to you whether you add one to everything, or simply do that when exporting the VCF using the position_transform parameter. If not, you will need to ensure no site at pos 0, and have ts.sequence length as L+1

[juleslr13]

Thank you both.

After some investigation, I realized that what I initially thought were mutation miscalls/ Position issues in the VCF might actually be mapping artifacts of my simulated reads due to simulating mutations in positions corresponding to missing values (first ~5 000 000 positions).

Setting up a RateMap that blocks or minimizes recombination and mutation rates over the N regions seems to be efficient to avoid both false mutation generation and downstream mapping issues. However, my pipeline relies on using trees.alignments(), and this function does not handle missing data and raises error #1896 if there are any intervals without trees (which happens if you use np.nan or leave out regions in the RateMap).

@hyanwong, I appreciate your suggestion about simulating all regions (including those that are N) and masking them afterwards. My first thought was that in my pipeline, this may cause problems further downstream. If I simulate mutations in NNN regions and then generate reads for mapping, these reads will be aligned to a damaged reference with stretches of Ns, which can lead to mapping errors-especially since my simulation aims to model damaged reads. But your most recent suggestion, would resolve that.
I’ll try calling ts.keep_intervals() (or ts.delete_intervals()) immediately after generation to mask out the N regions and avoid downstream mapping errors; I haven’t tested it yet, but I’ll report back soon.

[hyanwong]

The keep_intervals and delete_intervals functions will create missing data regions, and therefore might not be compatible with using alignments. You may need to iterate over the variants instead (this could involved skipping non-variable sites).

If you really want to visit each position in the sequence, you could define a site for each integer position, but only add mutations in non-missing sites. I think this should work with the variants iterator (and the less efficient haplotypes iterator)

import msprime
import numpy as np

ts = msprime.sim_ancestry(10, sequence_length=100)
ts = msprime.sim_mutations(ts, rate=0.01)  # Add variable sites at a few positions
tables = ts.dump_tables()
tables.delete_intervals([[0, 10]])

# Add sites at all integer positions, if they don't exist already
# Just for illustration, use "." as the ancestral state (normally you might use the ref seq)
required_positions = np.ones(int(ts.sequence_length))
required_positions[ts.sites_position.astype(int)] = 0
for pos in np.where(required_positions)[0]:
    tables.sites.add_row(position=pos, ancestral_state=".")
tables.sort()
ts_with_all_sites = tables.tree_sequence()


#for a in ts_with_all_sites.alignments():  # will not work because of missing data
#    print(a)

# iterate over all sites (including ones with missing data)
for v in ts_with_all_sites.variants():
    alleles = np.concatenate((v.alleles, ["N"]))
    print("".join(alleles[v.genotypes]))

# Or iterate over samples (haplotypes): this is less efficient than over sites, however
for h in ts_with_all_sites.haplotypes():
    print(h)

But ideally we would fix #1896 so that alignments work with missing data.