Progress stuck at findAlleles

[zongxiuXu]

Hello, I googled and couldn't find any solution, so I can only ask for help here. Could someone know the reason?

I installed miniconda in my own permission directory since I use the PBS system to run my commands on the server. And I have already updated my MiXCR using conda update mixcr.
Almost all commands of MiXCR work well according to the manual, but I have encountered difficulties in the first process of SHM analysis——findAlleles.
My data is bulk IgH sequencing of 20 mouse tissues of the same strain, and their reads range from 1 million to 2.5 million. Therefore I executed:

mixcr -Xmx40G analyze generic-bcr-amplicon --species mmu --rna --rigid-left-alignment-boundary --rigid-right-alignment- boundary --tag-pattern "^N{4}N{22}(R1:*)\^N{4}N{20}(R2:*)" --assemble-clonotypes-by {CDR1Begin:FR4End} --verbose /home/Data_save/Antibody_repertoire_sequencing/${name}_R1.fastq.gz /home/Data_save/Antibody_repertoire_sequencing/${name}_R2.fastq.gz ${analy}${name}

Very successful and the align success rate is very high. And then:

mixcr findAlleles --report ${SHM}.findAlleles.report.txt --export-alleles-mutations ${SHM}_alleles.tsv --export-library ${SHM}_alleles.json --output-template {file_dir_path}/{file_name}.reassigned.clns ${analy}*.clns

This process is running for nearly two hours, but I looked at the log and found that it has been looping Searching for V alleles: 40.2%.

The whole log is as follows:
Grouping by the same J gene: 0%
Grouping by the same J gene: 18.1% ETA: 00:00:09
Grouping by the same J gene: 29.6% ETA: 00:00:06
Grouping by the same J gene: 40.2% ETA: 00:00:05
Searching for J alleles: 6.3%
Searching for J alleles: 40.2% ETA: 00:00:01
Grouping by the same V gene: 8.8%
Grouping by the same V gene: 21.9% ETA: 00:00:05
Grouping by the same V gene: 32.8% ETA: 00:00:06
Searching for V alleles: 10.1%
Searching for V alleles: 32% ETA: 00:00:03
Searching for V alleles: 40.2% ETA: 00:14:33
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
Searching for V alleles: 40.2%
...

I guessed that it was stuck because the java memory was not large enough, so I tried the following command:
mixcr -Xmx400G findAlleles ...
and
mixcr findAlleles -t 48 ...
and
mixcr -Xmx400G findAlleles -t 48 ...
But none of those work, and the log output is all stuck at 40.2%.
By the way, my pbs file is setting: PBS -l nodes=2:ppn=24.
Now I wonder is it because I have too many files to analyze, or is the file too large? Or is my allocated memory still not enough? Or is this simply a bug?
Very much hoping someone can help. Sincere thanks!!!!!!!!!!

[gnefedev]

Hello, thanks for your report.

I think that program wasn't stuck, but progress is not very informative. You encountered a situation where some genes are much more represented in data than others and they will be processed much slower. Because of that progress pass is fast at the beginning and gets slower.

Nevertheless, we rewrite progress, it should be more informative now. If you will encounter this situation on the new version, try to run the command with --verbose argument, you will see more details in the logs.

You can download the new version by link https://link.milaboratory.com/software/mixcr/mixcr-develop.zip. Lets us know about results.

[zongxiuXu]

Sincerely thank for your reply! As long as it's not an out-of-memory issue it's fine, in fact I ran the entire SHM pipeline using 2 of the 20 seq data and they worked fine.

As for the new version, I downloaded https://link.milaboratory.com/software/mixcr/mixcr-develop.zip and created a new environment in miniconda for it. Execute conda install openjdk to provide a java environment. And then I run:

java -jar /home/mixcr4_test/mixcr.jar findAlleles -t 48 --verbose --report ${SHM}.findAlleles.report.txt --export-alleles-mutations ${SHM}_alleles.tsv --export-library ${SHM}_alleles.json --output-template ${SHM}{file_name}.reassigned.clns ${analy}*.clns

And the pbs command unchanged: PBS -l nodes=2:ppn=24

I was pleasantly surprised to see a more detailed log so that users can't feel stuck anymore.

The log is as follows:

Available CPU: 28, used CPU: unspecified. Available memory: 257679 Mb, used memory: 30688 Mb
Step 1 of 7: count diversity for dataset: 0%
Step 1 of 7: count diversity for dataset: 10.2% ETA: 00:00:08
Step 1 of 7: count diversity for dataset: 28.2% ETA: 00:00:04
Step 1 of 7: count diversity for dataset: 44.2% ETA: 00:00:03
Step 1 of 7: count diversity for dataset: 60.6% ETA: 00:00:02
Step 1 of 7: count diversity for dataset: 76.1% ETA: 00:00:01
Step 1 of 7: count diversity for dataset: 90.5% ETA: 00:00:00
Step 2 of 7: grouping by the same J gene: 0%
Step 2 of 7: grouping by the same J gene: 24.2% ETA: 00:00:06
Step 2 of 7: grouping by the same J gene: 43.1% ETA: 00:00:03
Step 2 of 7: grouping by the same J gene: 60.4% ETA: 00:00:02
Step 2 of 7: grouping by the same J gene: 77.9% ETA: 00:00:01
Step 2 of 7: grouping by the same J gene: 95.7% ETA: 00:00:00
Step 2 of 7: Searching for J alleles: 0%
Found alleles mutations: EMPTY from IGHJ100, total clones for search: 91343
Step 2 of 7: Searching for J alleles: 25% ETA: 00:00:09
Found alleles mutations: EMPTY from IGHJ400, total clones for search: 124190
Found alleles mutations: EMPTY from IGHJ300, total clones for search: 126084
Step 2 of 7: Searching for J alleles: 75% ETA: 00:00:00
Found alleles mutations: EMPTY from IGHJ200, total clones for search: 238651
Step 3 of 7: grouping by the same V gene: 0%
Step 3 of 7: grouping by the same V gene: 14.8% ETA: 00:00:05
Step 3 of 7: grouping by the same V gene: 31% ETA: 00:00:04
Step 3 of 7: grouping by the same V gene: 47.8% ETA: 00:00:03
Step 3 of 7: grouping by the same V gene: 65.5% ETA: 00:00:01
Step 3 of 7: grouping by the same V gene: 82% ETA: 00:00:01
Step 3 of 7: Searching for V alleles: 0%
Found alleles mutations: EMPTY from IGHV5-1600, total clones for search: 673
Found alleles mutations: EMPTY from IGHV5-1500, total clones for search: 828
Found alleles mutations: EMPTY from IGHV1-200, total clones for search: 209
Found alleles mutations: EMPTY from IGHV5-1200, total clones for search: 1236
Found alleles mutations: from IGHV3-300, total clones for search: 14
Found alleles mutations: EMPTY from IGHV3-400, total clones for search: 28
Found alleles mutations: EMPTY from IGHV2-400, total clones for search: 81
Found alleles mutations: EMPTY from IGHV3-500, total clones for search: 65
Found alleles mutations: EMPTY from IGHV4-100, total clones for search: 41
Found alleles mutations: EMPTY from IGHV2-600, total clones for search: 86
Found alleles mutations: from IGHV7-400, total clones for search: 12
Found alleles mutations: EMPTY from IGHV8-500, total clones for search: 31
Found alleles mutations: EMPTY from IGHV8-200, total clones for search: 36
Found alleles mutations: EMPTY from IGHV7-100, total clones for search: 79
Found alleles mutations: EMPTY from IGHV7-600, total clones for search: 94
Found alleles mutations: EMPTY from IGHV2-900, total clones for search: 146
Found alleles mutations: EMPTY from IGHV2-500, total clones for search: 165
Found alleles mutations: EMPTY from IGHV5-200, total clones for search: 159
Found alleles mutations: EMPTY from IGHV2-300, total clones for search: 178
Found alleles mutations: EMPTY from IGHV6-300, total clones for search: 227
Found alleles mutations: EMPTY from IGHV3-800, total clones for search: 309
Found alleles mutations: EMPTY from IGHV2-200, total clones for search: 441
Found alleles mutations: EMPTY from IGHV8-800, total clones for search: 456
Found alleles mutations: EMPTY from IGHV3-600, total clones for search: 2004
Found alleles mutations: EMPTY from IGHV7-300, total clones for search: 1359
Step 3 of 7: Searching for V alleles: 25.5% ETA: 00:00:02
Found alleles mutations: EMPTY from IGHV5-400, total clones for search: 2973
Found alleles mutations: EMPTY from IGHV3-100, total clones for search: 334
Found alleles mutations: from IGHV1-1200, total clones for search: 19
Found alleles mutations: EMPTY from IGHV1-1100, total clones for search: 197
Found alleles mutations: EMPTY from IGHV1-2300, total clones for search: 63
Found alleles mutations: from IGHV1-4900, total clones for search: 10
Found alleles mutations: EMPTY from IGHV1-2600, total clones for search: 622
Found alleles mutations: EMPTY from IGHV1-4300, total clones for search: 439
Found alleles mutations: EMPTY from IGHV1-3600, total clones for search: 511
Found alleles mutations: EMPTY from IGHV1-6100, total clones for search: 528
Found alleles mutations: ST407GST428C from IGHV1-3700, total clones for search: 792
Found alleles mutations: EMPTY from IGHV1-5600, total clones for search: 640
Found alleles mutations: EMPTY from IGHV1-5000, total clones for search: 1205
Found alleles mutations: EMPTY from IGHV1-5900, total clones for search: 955
Found alleles mutations: EMPTY from IGHV1-3100, total clones for search: 1286
Found alleles mutations: EMPTY, SA440T from IGHV1-5200, total clones for search: 1418
Found alleles mutations: EMPTY from IGHV1-3400, total clones for search: 1746
Found alleles mutations: EMPTY from IGHV1-7400, total clones for search: 1701
Found alleles mutations: EMPTY, SA184TSA188TST203ASC209ASA225GSG226ASC227ASG232ASA233CSG266ASC327ASC343GSC350GSA356CSA38
2G, SA184TSA188TST203ASC209ASA225GSG226CSC228ASG232ASG236ASG266ASC327ASC343GSC350GSA356CSA382G, SA184TSA188TSA198GSC209A
SG223ASA225GSG226TSG231ASG232ASG236ASG266ASC327ASC343GSC350GSA356CSA382G from IGHV1-8300, total clones for search: 508
Found alleles mutations: SG420TST426C from IGHV1-2200, total clones for search: 2130
Step 3 of 7: Searching for V alleles: 51% ETA: 00:00:05
Found alleles mutations: ST358G, EMPTY from IGHV1-6900, total clones for search: 2092
Found alleles mutations: SG432C, SA440T, SA352GST371CSC379TSC393GSC400GSA406CSA440T from IGHV1-62-300, total clones for
search: 2030
Found alleles mutations: EMPTY from IGHV1-1800, total clones for search: 2394
Found alleles mutations: EMPTY, SG383ASA395TSG407CSA416GSA429G from IGHV5-9-100, total clones for search: 2912
Found alleles mutations: EMPTY, SA234CSC249TSC252TST253GSA254CSA259CSG270ASC276TSG281CSC288TST291ASA311GSC313ASA319TSC32
1TSG326ASA328GSG353AST364AST372ASG378ASC396TSG402ASC405TSA406GSA407CSC411ASA432C from IGHV1-1900, total clones for sear
ch: 2207
Possible poly-zygote, searching alleles separately for genes [IGHV1-19, IGHV1-67]
Found alleles mutations: SA431G from IGHV1-58*00, total clones for search: 2313

But also generated err file:

4.3.1-12-develop; built=Tue Mar 28 20:54:28 CST 2023; rev=2a4efedf0a; lib=repseqio.v2.2
picocli.CommandLine$ExecutionException: Error while running command findAlleles java.lang.RuntimeException: java.lang.Nu
llPointerException
at com.milaboratory.mixcr.cli.Main.registerExceptionHandlers$lambda-13(SourceFile:322)
at picocli.CommandLine.execute(CommandLine.java:2088)
at com.milaboratory.mixcr.cli.Main.main(SourceFile:102)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at cc.redberry.pipe.blocks.ParallelProcessor.take(ParallelProcessor.java:246)
at cc.redberry.pipe.util.CountingOutputPort$Companion$wrap$1.take(CountingOutputPort.kt:35)
at com.milaboratory.o.zB.take(SourceFile:24)
at cc.redberry.pipe.CUtils$OPIterator.hasNext(CUtils.java:256)
at kotlin.sequences.SequencesKt___SequencesKt.toCollection(_Sequences.kt:787)
at kotlin.sequences.SequencesKt___SequencesKt.toMutableList(_Sequences.kt:817)
at kotlin.sequences.SequencesKt___SequencesKt$sortedWith$1.iterator(_Sequences.kt:638)
at kotlin.collections.MapsKt__MapsKt.putAll(Maps.kt:426)
at kotlin.collections.MapsKt__MapsKt.toMap(Maps.kt:594)
at kotlin.collections.MapsKt__MapsKt.toMap(Maps.kt:588)
at com.milaboratory.mixcr.alleles.AllelesBuilder.a(SourceFile:209)
at com.milaboratory.mixcr.cli.CommandFindAlleles.run1(SourceFile:342)
at com.milaboratory.mixcr.cli.MiXCRCommandWithOutputs.run0(SourceFile:69)
at com.milaboratory.mixcr.cli.MiXCRCommand.run(SourceFile:37)
at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
at com.milaboratory.mixcr.cli.Main.registerLogger$lambda-27(SourceFile:430)
at picocli.CommandLine.execute(CommandLine.java:2078)
... 1 more
Caused by: java.lang.NullPointerException
at com.milaboratory.o.hN.a(SourceFile:75)
at com.milaboratory.o.hN.a(SourceFile:30)
at com.milaboratory.o.iq.process(SourceFile:86)
at cc.redberry.pipe.blocks.ParallelProcessor$Worker.run(ParallelProcessor.java:304)
at java.base/java.lang.Thread.run(Thread.java:829)

And unfortunately there is no output.

Hope you guys can solve this problem.

A few more words. I have a very good experience with MiXCR4. There are more details and parameters, a simpler process and more pretty results. I can feel the innovation and hard work. Looking forward to a perfect mixcr4! (Hope that the SHM R plot function in guides can be implemented in the future haha)
By the way, I found some tiny text errors while reading the manual and I can provide them if needed.
Thanks!

[gnefedev]

Glad to hear your review!

I found a possible reason for the error and fix it. You can download the fixed version by the same link https://link.milaboratory.com/software/mixcr/mixcr-develop.zip

Off course, share with us the text errors that you found.

[zongxiuXu]

Sorry for the long wait! Our server has been quite busy recently, and my task has been queued for a long time.
As in the previous reply, I downloaded and installed the latest version of MiXCR. And then run the command as follows:

java -jar /home/mixcr4_test/mixcr.jar findAlleles -t 48 --verbose --report ${SHM}.findAlleles.report.txt --export-alleles-mutations ${SHM}_alleles.tsv --export-library ${SHM}_alleles.json --output-template ${SHM}{file_name}.reassigned.clns ${analy}*.clns

Great! The program ended without error after about three hours, and the output all look fine.The following is part of the log (the original file is large):

============== FindAlleles Report ==============
Analysis time: 168.99m
Clones score delta stats:
size: 1182904
sum: 1.943315316022339E8
min: -29.0
max: 870.0
avg: 164.2834343296125
quadratic mean: 206.95931394327127
std deviation: 125.86947297161663
Clones count with no change of score: 259937
Clones count with negative score change: 12
Found alleles on first try: 132
Zygotes: {1=44, 48=1, 4=4, 2=6, 11=3, 9=1, 7=2, 17=2, 43=1, 46=1, 5=3, 33=2, 116=1, 3=3, 10=2, 36=2, 13=1, 37=2, 18=1, 16=2, 90=1, 15=1, 28=1, 57=1, 20=2, 158=1, 192=1, 110=2, 51=1, 38=1, 78=1, 88=1, 44=1, 25=1, 8=1, 12=1}
Not enough information for allele search: [IGHV1-12, IGHV1-16, IGHV1-24, IGHV1-33, IGHV1-62-1, IGHV11-2, IGHV15-2, IGHV8-9]
Found alleles: 1776
De novo alleles: 1776
Known variant from library: 0
Not enough info for search, but aligned on known allele: 0
Not changed genes: 102
Not changed after search: 94
No clones to search: 5
Have clones to search, but not enough info for search and can't be aligned on known variant: 3
Removed genes: 521
No top hits in result files: 0
No top hits in source files: 0
Can be aligned on other gene: 6
Not represented in source files: 515
Found alleles that don't fit well to data: {}

I'll mark your reply as solution after I get the output passed without issue through the other steps of SHM.

---

Glad to be able to help, so how do I send you those text errors? Could you leave an email address if possible?

Besides, I have another question:
Although my 20 samples come from the same background C57BL6, they are actually from 20 separate individuals. This number is relatively large, so I think it may not be appropriate to put 20 samples together to do findAlleles considering the impact of germline mutations. Then I want to ask that can I skip this step and go directly to other steps in the SHM analysis process for my data situation, that is, use the data generated by analyze to directly findShmTrees.

To be honest, I don't understand the SHM very well, so if you can answer my question I would be very very grateful!!!

Thanks!

[gnefedev]

Yes, it definitely should be run by individuals separately.

findAlleles is mandatory to run before SHM, otherwise, allele mutations will be counted as evidence of an existing common ancestor. I understand that there are maybe some types of data that already have resolved alleles, but in this case findAlleles should not find anything (so it's harmless, especially comparing it to running findShmTrees directly).

Just in case, you should run findShmTrees also on each individual separately.

Please, let me know about the results after running on separate individuals, I'm curious to see, will be different alleles found on different mice. You can run exportReportsTable to see general stats for all result files in one file.

About text errors, you can send them to [email protected] or make a pull request to https://github.com/milaboratory/docs