Parallelize Ranked Bray-Curtis Calculation in ExtractCore Function

[jibarozzo]

Issue: Parallelize Ranked Bray-Curtis Calculation in ExtractCore Function

Description:
The ranked Bray-Curtis (BC) portion of the ExtractCore function is computationally intensive and does not utilize all available cores on the HPCC. This issue proposes parallelizing the BC calculations to improve performance.

---

Problem

• The current implementation processes OTUs sequentially, leading to long runtimes for datasets with many OTUs.
• The HPCC's available cores are underutilized during execution.

---

Proposed Solution

1. Parallelize calculate_bc():

   • Use the parallel or doParallel package to distribute BC calculations across multiple cores.
   • Focus on the loop where ranked OTUs are added iteratively to the BC matrix.

2. Refactor Ranked BC Logic:

   • Introduce a new helper function, rank_bc(), to handle the ranked OTU calculations.
   • Optimize the loop to stop early if BC contributions flatline (e.g., after the first 1000 OTUs).

---

Discussion Points

• Should parallelization focus on:
  • The calculate_bc() function itself?
  • The loop that iterates over ranked OTUs?
  • Both?
• How to handle early termination when BC contributions plateau?

---

Next Steps

• Implement parallelization using parallel or doParallel.
• Refactor ranked BC logic into rank_bc().
• Add early termination logic for flatlining BC contributions.
• Benchmark performance improvements on HPCC.

---

Collaborators:

• @jibarozzo
• @B-Kristy