Bug: Overflow error when using Poisson law for bdgcmp

[callum-b]

I'm trying to run bdgcmp on some spike-in normalised bedGraphs I've generated used MACS3 tools. Here is the command line I'm running:
macs3 bdgcmp -m qpois -t DATA/BED/TEAD1/TEAD1_T1_pos_1_pileup_GRCh38_sort_spikeBDGP6.32_pe.bedGraph -c DATA/BED/TEAD1/Input_T1_pos_pe_pileup_sort_norm10M_nozero.bedGraph -o DATA/BED/TEAD1/TEAD1_T1_pos_1_pileup_GRCh38_sort_spikeBDGP6.32_pe_qval.bedGraph

Here's the output from MACS3:

INFO
@ 21 Oct 2024 17:49:30: [38 MB] Read and build treatment bedGraph...
INFO
@ 21 Oct 2024 17:49:32: [65 MB] Read and build control bedGraph...
INFO
@ 21 Oct 2024 17:49:35: [96 MB] Build ScoreTrackII...
INFO
@ 21 Oct 2024 17:49:38: [154 MB] Calculate scores comparing treatment and control by 'qpois'...
Traceback (most recent call last):
File "MACS3/Signal/ScoreTrack.pyx", line 67, in MACS3.Signal.ScoreTrack.get_pscore
File "src/cykhash/maps/map_impl.pxi", line 2448, in cykhash.khashmaps.PyObjectMap.__getitem__
File "src/cykhash/maps/map_impl.pxi", line 2445, in cykhash.khashmaps.PyObjectMap.cget
KeyError: (-2147483648, 0.009999999776482582)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bioinfo/src/MACS/MACS-v3.0.0/MACS-v3.0.0_venv/bin/macs3", line 1028, in <module>
main()
File "/usr/local/bioinfo/src/MACS/MACS-v3.0.0/MACS-v3.0.0_venv/bin/macs3", line 65, in main
run( args )
File "/usr/local/bioinfo/src/MACS/MACS-v3.0.0/MACS-v3.0.0_venv/lib/python3.11/site-packages/MACS3/Commands/bdgcmp_cmd.py", line 72, in run
sbtrack.change_score_method( ord('q') )
File "MACS3/Signal/ScoreTrack.pyx", line 354, in MACS3.Signal.ScoreTrack.ScoreTrackII.change_score_method
File "MACS3/Signal/ScoreTrack.pyx", line 371, in MACS3.Signal.ScoreTrack.ScoreTrackII.change_score_method
File "MACS3/Signal/ScoreTrack.pyx", line 407, in MACS3.Signal.ScoreTrack.ScoreTrackII.compute_pvalue
File "MACS3/Signal/ScoreTrack.pyx", line 69, in MACS3.Signal.ScoreTrack.get_pscore
File "MACS3/Signal/Prob.pyx", line 209, in MACS3.Signal.Prob.poisson_cdf
OverflowError: can't convert negative value to npy_uint32

I tried running the same command on a subset of the bedGraphs (chr15) and got the same behaviour, so I don't think it's tied to the size of the file/number of reads, which was my first thought given the Overflow error. These files can be downloaded here and here in gzipped format.

bdgcmp does however work when using -m subtract, so it probably isn't coming from the data files I'm processing.

What's happening with the Poisson law in bdgcmp? Why is it failing?

System

• OS: Ubuntu 20.04.6 LTS
• Python version: 3.10.14
• Numpy version: 1.23.5
• MACS Version: 3.0.1

[taoliu]

@callum-b May I ask why the values in the treatment file 'TEAD1_T1_pos_1_pileup_GRCh38_sort_spikeBDGP6.32_pe.bedGraph' is extremely high? The value is either 0 or larger than 1e+7.

$ cut -f 4 -d" " TEAD1_T1_pos_1_pileup_GRCh38_sort_spikeBDGP6.32_pe_chr15.bedGraph | sort -u -g | head -10
0
1.21207e+07
2.42414e+07
3.63621e+07
4.84827e+07
6.06034e+07
7.27241e+07
8.48448e+07
9.69655e+07
1.09086e+08

But the control file 'Input_T1_pos_pe_pileup_sort_norm10M_nozero.bedGraph' contains value ranging from 0.01 to 8.5. Before bdgcmp, the two pileup files need to be normalized.

[callum-b]

That was an error on my part, not sure how I missed it. I reran the normalisation for that file and have numbers which are much more reasonable, and the qpois calculations work now.

Is an Overflow error the expected behaviour from this?

[taoliu]

@callum-b The overflow error may be due to the big difference between treatment and control pileup.