Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Good practice for Reference-Mapped Short Read Genomes? #1405

Open
dylandebaun opened this issue Jun 11, 2024 · 2 comments
Open

Good practice for Reference-Mapped Short Read Genomes? #1405

dylandebaun opened this issue Jun 11, 2024 · 2 comments

Comments

@dylandebaun
Copy link

Hello,
I have some short-read genomes that were assembled via reference mapping (different references) that I would like to align to each other. I have consensus sequences with masked reference sites. We are discussing ways forward with this (since hard masking is inadvisable), either:

  1. keeping these reference sites soft-masked along with soft-masked repetitive regions as input in cactus (since they hold some alignment information) OR

  2. removing these regions from the fasta prior to cactus alignment (since they are not all repetitive regions and are likely incorrect as they were just 'carried over' from the reference [different species]).

Any insight on this? Is cactus treating soft masked as a repeat exclusively? Most papers I've seen only align denovo assemblies.
@glennhickey
Copy link
Collaborator

Sorry, I'm not very familiar with this type of data. If you want these regions in the alignment, I'd suggest leaving them and not masking them any differently than the rest of the assembly. If you don't want them in the alignment, then hardmasking may be simpler than cutting them out, especially if cutting them out will result in tons of tiny contigs.

Someone was asking recently about masking after aligning, which seems a bit related to what you may want to do as well, but I'm not aware of any tools that do this directly.

@dylandebaun
Copy link
Author

dylandebaun commented Jun 21, 2024 •
edited
Loading

Using genomes with a high proportion of Ns causes the aligner to take much longer and is not using the GPU resources. What stage might be this be stuck on- is there any work around to speed this up like a normal run? Also, is it possible to add more logging information to the segalign step?

I'm running this using kubernetes scheduler, requesting a single machine.
Command:

cactus-blast /data/jobStore/1 /data/scripts/preparesteps_mask.txt /data/steps-output-thammask/Anc4.paf --root Anc4 --gpu 2 --logFile /data/steps-output-thammask/logs/blast-Anc4.log --lastzCores 20 --lastzMemory 200G;

resources:
          limits:
            memory: 200Gi
            cpu: 20
            nvidia.com/gpu: "2"
            ephemeral-storage: 200Gi
          requests:
            memory: 200Gi
            cpu: 20
            nvidia.com/gpu: "2"
            ephemeral-storage: 200Gi
Screenshot 2024-06-21 at 9 01 56 AM 
Wed Jun 19 18:31:03 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76                 Driver Version: 550.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10                     Off |   00000000:01:00.0 Off |                    0 |
|  0%   35C    P8             21W /  150W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A10                     Off |   00000000:A1:00.0 Off |                    0 |
|  0%   34C    P8             20W /  150W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
start
[2024-06-19T18:31:04+0000] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil
[2024-06-19T18:31:04+0000] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.human2bytes()" instead."
[2024-06-19T18:31:04+0000] [MainThread] [W] [toil.lib.humanize] Deprecated toil method.  Please use "toil.lib.conversions.human2bytes()" instead."
[2024-06-19T18:31:04+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /home/cactus/cactus_env/bin/cactus-blast /data/jobStore/1 /data/scripts/preparesteps_mask.txt /data/steps-output-thammask/Anc4.paf --root Anc4 --gpu 2 --logFile /data/steps-output-thammask/logs/blast-Anc4.log --lastzCores 20 --lastzMemory 200G
[2024-06-19T18:31:04+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: 28a6e2cbecb5e954298f60d3ede2bdf7218f11d7
[2024-06-19T18:31:04+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/steps-output-thammask/Masked_Tham_Ep_Hifiasm.scaffold.fasta
[2024-06-19T18:31:08+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/steps-output-thammask/Thamnosophis-epistibes-cf-RAN42556-iter-02-softmask-final.fa.HM
[2024-06-19T18:31:08+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/steps-output-thammask/Thamnosophis_lateralis_RAX10040_iter-02-softmask-final.fa.HM
[2024-06-19T18:31:08+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/steps-output-thammask/Thamnosophis_stumpfii_RAX2284_iter-02-softmask-final.fa.HM
[2024-06-19T18:31:08+0000] [MainThread] [I] [toil] Running Toil version 6.0.0-0e2a07a20818e593bfdfde3cc51ca4ad809fde96 on host testround3-4h2js.
[2024-06-19T18:31:08+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2024-06-19T18:31:08+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_then_make_paf_alignments' kind-sanitize_then_make_paf_alignments/instance-ziwrwgho v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-o_n1n0cy v1 with job batch system ID: 2 and disk: 11.8 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-pi0p2d2c v1 with job batch system ID: 3 and disk: 11.8 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-f2ze9nne v1 with job batch system ID: 4 and disk: 11.8 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-jcj0ce5c v1 with job batch system ID: 5 and disk: 11.8 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:09.732698: Running the command: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/7539/2958/tmpdtl0detq/Thamnosophis-epistibes-cf.fa Thamnosophis-epistibes-cf"
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:09.740402: Running the command: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/cfab/4a5f/tmpmk1wkgcb/Thamnosophis-stumpfii.fa Thamnosophis-stumpfii"
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:09.881423: Running the command: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/a826/d71f/tmpsa1qs8zo/Thamnosophis-lateralis.fa Thamnosophis-lateralis"
[2024-06-19T18:31:09+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:09.886829: Running the command: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/8bd1/c730/tmp521kzhwz/Tham-Ep.fa Tham-Ep"
[2024-06-19T18:31:26+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:26.076284: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/a826/d71f/tmpsa1qs8zo/Thamnosophis-lateralis.fa Thamnosophis-lateralis" in 16.1905 seconds and 461.2 Mi memory
[2024-06-19T18:31:26+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:26.079083: Running the command: "cactus_analyseAssembly Thamnosophis-lateralis.sanitized.fa"
[2024-06-19T18:31:26+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:26.144167: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/7539/2958/tmpdtl0detq/Thamnosophis-epistibes-cf.fa Thamnosophis-epistibes-cf" in 16.405 seconds and 461.2 Mi memory
[2024-06-19T18:31:26+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:26.146800: Running the command: "cactus_analyseAssembly Thamnosophis-epistibes-cf.sanitized.fa"
[2024-06-19T18:31:33+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:33.006504: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/cfab/4a5f/tmpmk1wkgcb/Thamnosophis-stumpfii.fa Thamnosophis-stumpfii" in 23.2553 seconds and 461.2 Mi memory
[2024-06-19T18:31:33+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:33.009209: Running the command: "cactus_analyseAssembly Thamnosophis-stumpfii.sanitized.fa"
[2024-06-19T18:31:33+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:33.254051: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/55a7156da3105651aeeda9fb3d7b81e4/8bd1/c730/tmp521kzhwz/Tham-Ep.fa Tham-Ep" in 23.3628 seconds and 461.2 Mi memory
[2024-06-19T18:31:33+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:33.256612: Running the command: "cactus_analyseAssembly Tham-Ep.sanitized.fa"
[2024-06-19T18:31:40+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:40.175424: Successfully ran: "cactus_analyseAssembly Thamnosophis-lateralis.sanitized.fa" in 14.0905 seconds and 461.2 Mi memory
[2024-06-19T18:31:40+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:40.628738: Successfully ran: "cactus_analyseAssembly Thamnosophis-epistibes-cf.sanitized.fa" in 14.4721 seconds and 461.3 Mi memory
[2024-06-19T18:31:41+0000] [Thread-4  ] [I] [toil.statsAndLogging] Got message from job at time 06-19-2024 18:31:41: Assembly stats for Thamnosophis-lateralis: Input-sample: Thamnosophis-lateralis.sanitized.fa Total-sequences: 577 Total-length: 1797975162 Proportion-repeat-masked: 0.708188 ProportionNs: 0.532739 Total-Ns: 957851720 N50: 198535865 Median-sequence-length: 54030 Max-sequence-length: 359040883 Min-sequence-length: 12052

[2024-06-19T18:31:41+0000] [Thread-4  ] [I] [toil.statsAndLogging] Got message from job at time 06-19-2024 18:31:41: Assembly stats for Thamnosophis-epistibes-cf: Input-sample: Thamnosophis-epistibes-cf.sanitized.fa Total-sequences: 577 Total-length: 1795754216 Proportion-repeat-masked: 0.701499 ProportionNs: 0.534128 Total-Ns: 959163399 N50: 198219644 Median-sequence-length: 54042 Max-sequence-length: 358518777 Min-sequence-length: 12052

[2024-06-19T18:31:47+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:47.366349: Successfully ran: "cactus_analyseAssembly Tham-Ep.sanitized.fa" in 14.1063 seconds and 461.2 Mi memory
[2024-06-19T18:31:47+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:47.416605: Successfully ran: "cactus_analyseAssembly Thamnosophis-stumpfii.sanitized.fa" in 14.4025 seconds and 461.2 Mi memory
[2024-06-19T18:31:48+0000] [Thread-4  ] [I] [toil.statsAndLogging] Got message from job at time 06-19-2024 18:31:48: Assembly stats for Thamnosophis-stumpfii: Input-sample: Thamnosophis-stumpfii.sanitized.fa Total-sequences: 577 Total-length: 1799093806 Proportion-repeat-masked: 0.707641 ProportionNs: 0.532681 Total-Ns: 958343539 N50: 198681329 Median-sequence-length: 54030 Max-sequence-length: 359330886 Min-sequence-length: 12052

[2024-06-19T18:31:48+0000] [MainThread] [I] [toil.leader] Issued job 'make_paf_alignments' kind-make_paf_alignments/instance-co9ap5he v1 with job batch system ID: 6 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:48+0000] [Thread-4  ] [I] [toil.statsAndLogging] Got message from job at time 06-19-2024 18:31:48: Assembly stats for Tham-Ep: Input-sample: Tham-Ep.sanitized.fa Total-sequences: 577 Total-length: 1796722112 Proportion-repeat-masked: 0.574171 ProportionNs: 0.000019 Total-Ns: 34800 N50: 198345250 Median-sequence-length: 54030 Max-sequence-length: 358678011 Min-sequence-length: 12052

[2024-06-19T18:31:49+0000] [MainThread] [I] [toil.leader] Issued job 'make_ingroup_to_outgroup_alignments_0' kind-make_ingroup_to_outgroup_alignments_0/instance-75lk1x8l v1 with job batch system ID: 7 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:49+0000] [MainThread] [I] [toil.leader] Issued job 'make_ingroup_to_outgroup_alignments_0' kind-make_ingroup_to_outgroup_alignments_0/instance-nbwg8mf_ v1 with job batch system ID: 8 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:49+0000] [MainThread] [I] [toil.leader] Issued job 'make_chunked_alignments' kind-make_chunked_alignments/instance-hu249v40 v1 with job batch system ID: 9 and disk: 13.5 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:49+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:49.592260: Running the command: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmpz536jmzy /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmp5i5kqswj.tmp"
[2024-06-19T18:31:49+0000] [MainThread] [I] [toil.leader] Issued job 'make_chunked_alignments' kind-make_chunked_alignments/instance-ns3gu2l5 v1 with job batch system ID: 10 and disk: 13.5 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:49+0000] [MainThread] [I] [toil.leader] Issued job 'make_chunked_alignments' kind-make_chunked_alignments/instance-lqzjg5xd v1 with job batch system ID: 11 and disk: 13.5 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2024-06-19T18:31:50+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:50.122382: Running the command: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmplp23i37v /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmp70ks6gjw.tmp"
[2024-06-19T18:31:50+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:31:50.185833: Running the command: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmpbs9aagqj /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmplj5rha2l.tmp"
[2024-06-19T18:32:04+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:04.658058: Successfully ran: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmplp23i37v /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmp70ks6gjw.tmp" in 14.5301 seconds and 686.6 Mi memory
[2024-06-19T18:32:04+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:04.660804: Running the command: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmp371rx1y2 /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmp8wd9kveo.tmp"
[2024-06-19T18:32:08+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:08.476696: Successfully ran: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmpz536jmzy /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmp5i5kqswj.tmp" in 18.8819 seconds and 687.4 Mi memory
[2024-06-19T18:32:08+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:08.479495: Running the command: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmpu5wwii7b /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmpua36dxu_.tmp"
[2024-06-19T18:32:08+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:08.747630: Successfully ran: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmpbs9aagqj /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmplj5rha2l.tmp" in 18.5584 seconds and 686.7 Mi memory
[2024-06-19T18:32:08+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:08.750723: Running the command: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmpjci65en5 /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmpja9n52k3.tmp"
[2024-06-19T18:32:19+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:19.074252: Successfully ran: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmp371rx1y2 /tmp/55a7156da3105651aeeda9fb3d7b81e4/4eb7/8c76/tmp8wd9kveo.tmp" in 14.4049 seconds and 687.6 Mi memory
[2024-06-19T18:32:20+0000] [MainThread] [I] [toil.leader] Issued job 'run_lastz' kind-run_lastz/instance-xz2p4is3 v1 with job batch system ID: 12 and disk: 186.3 Gi, memory: 186.3 Gi, cores: 20, accelerators: [{'count': 2, 'kind': 'gpu', 'api': 'cuda', 'brand': 'nvidia'}], preemptible: False
[2024-06-19T18:32:22+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:22.995111: Successfully ran: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmpu5wwii7b /tmp/55a7156da3105651aeeda9fb3d7b81e4/b36b/7715/tmpua36dxu_.tmp" in 14.5126 seconds and 687.0 Mi memory
[2024-06-19T18:32:23+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:23.173862: Successfully ran: "faffy chunk -c 6000000000 -o 10000 --dir /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmpjci65en5 /tmp/55a7156da3105651aeeda9fb3d7b81e4/bc12/018c/tmpja9n52k3.tmp" in 14.42 seconds and 687.2 Mi memory
[2024-06-19T18:32:24+0000] [MainThread] [I] [toil.leader] Issued job 'run_lastz' kind-run_lastz/instance-s95qzntu v1 with job batch system ID: 13 and disk: 186.3 Gi, memory: 186.3 Gi, cores: 20, accelerators: [{'count': 2, 'kind': 'gpu', 'api': 'cuda', 'brand': 'nvidia'}], preemptible: False
[2024-06-19T18:32:24+0000] [MainThread] [I] [toil.leader] Issued job 'run_lastz' kind-run_lastz/instance-kuqubb62 v1 with job batch system ID: 14 and disk: 186.3 Gi, memory: 186.3 Gi, cores: 20, accelerators: [{'count': 2, 'kind': 'gpu', 'api': 'cuda', 'brand': 'nvidia'}], preemptible: False
[2024-06-19T18:32:24+0000] [MainThread] [I] [toil-rt] 2024-06-19 18:32:24.997646: Running the command: "run_segalign Thamnosophis-stumpfii_0.fa Thamnosophis-lateralis_0.fa --format=paf:minimap2 --step=2 --ambiguous=iupac,100,100 --ydrop=3000 --notransition --num_gpu 2"
[2024-06-19T19:31:11+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-19T20:31:11+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-19T21:31:12+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-19T22:31:13+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-19T23:31:13+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T00:31:13+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T01:31:14+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T02:31:15+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T03:31:17+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T04:31:18+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T05:31:19+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T06:31:19+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T07:31:19+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T08:31:20+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T09:31:22+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T10:31:23+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T11:31:24+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T12:31:26+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T13:31:27+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T14:31:29+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T15:31:31+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T16:31:32+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T17:31:33+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T18:31:35+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T19:31:35+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T20:31:36+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T21:31:38+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T22:31:38+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-20T23:31:38+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T00:31:39+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T01:31:39+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T02:31:39+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T03:31:40+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T04:31:40+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T05:31:41+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T06:31:41+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T07:31:41+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T08:31:42+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T09:31:42+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T10:31:42+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T11:31:43+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T12:31:43+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T13:31:44+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T14:31:44+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T15:31:44+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run
[2024-06-21T16:31:45+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 2 jobs are issued and waiting to run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@glennhickey @dylandebaun