feat: Add Option to Prevent Overlapping Replications of the Same Artifact in Harbor

[bupd]

Summary

This PR addresses a long-standing issue where overlapping replications of the same artifact can occur in Harbor, leading to unnecessary resource consumption and poor performance. By introducing a "Disable parallel replication" checkbox in the replication policy, it ensures that replication tasks for the same policy do not run in parallel, preventing bandwidth overload and queue backups, especially for large artifacts.

Similar Issues

• Replication job duplication #19937
• Maximum parallel replications for pull #20532
• Replication: avoid overlapping replication jobs #17269
• Replication taking longer each time #16656

Related Issues

• Replication to second harbor is extremely slow #15415
• replication job failed,but do not re-scheduled #7842

Why do we need this

1. Users have for long requesting this feature to have single replication execution per policy.
2. Avoid overlapping replications which is common in using scheduled replications.
3. Without this replication starts piling up.
4. And at some point it becomes un-handleable... leading to a snowball effect on bandwidth and the queue.
5. It makes no sense to run replication in parallel for same replication policy.
6. It is better to atleast have an option to set replication to single execution at a time per policy.
7. It is a huge problem when using bigger artifacts that are more than 80gb in size running parallel replication.

Changes Made

• Added Disable parallel replication Checkbox in Replication UI.
• Added Execution skipping.
• Updated Replication Policy.
• Updated worker.
• Added skip_if_running column in sql.

Screenshots

Todo

• Add Tests

Please indicate you've done the following:

• Well Written Title and Summary of the PR
• Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
• Accepted the DCO. Commits without the DCO will delay acceptance.
• Made sure tests are passing and test coverage is added if needed.
• Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.12%. Comparing base (c8c11b4) to head (a997836).
Report is 340 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #21347      +/-   ##
==========================================
+ Coverage   45.36%   46.12%   +0.75%     
==========================================
  Files         244      247       +3     
  Lines       13333    13883     +550     
  Branches     2719     2875     +156     
==========================================
+ Hits         6049     6403     +354     
- Misses       6983     7141     +158     
- Partials      301      339      +38     

Flag	Coverage Δ
unittests	46.12% <ø> (+0.75%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 491 files with indirect coverage changes

[wy65701436]

Thanks, @bupd, for your contribution! I believe this is a valuable new feature for Harbor that should follow the proposal process.

There are several questions we need to briefly discuss:

• What happens if the execution gets stuck in the 'running' status?
• How do we determine the relationship between the replication policy and the execution instance — by ID? And what happens if the policy content is updated after the execution begins?
• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

[Vad1mo]

This text is misleading, clearly explain what this option does, providing enough context, background, and result of enabling this future: e.g "Don't start a new replication until the previous one is terminated"

[Vad1mo]

bad wording, intention and outcome not clear

[Vad1mo]

bad wording

[tpoxa]

Thanks, @bupd, for your contribution! I believe this is a valuable new feature for Harbor that should follow the proposal process.

There are several questions we need to briefly discuss:

• What happens if the execution gets stuck in the 'running' status?

We were planning to use JobServiceMonitorClient to get the current state of job observations. As long observation says it's running - a new execution will be skipped. I believe we should fix the problem with stuck separately. What are your thoughts about that?

• How do we determine the relationship between the replication policy and the execution instance — by ID? And what happens if the policy content is updated after the execution begins?

Indeed PolicyID was never needed before. We can add it anyway to the job arguments so we can scan observations, parse job arguments and match policyID with the current.

• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

I believe even checking an artefact for existence produces an additonal HTTP request. So, we still adding some extra efficiency.

[Vad1mo]

• During the replication job (at least in Harbor-to-Harbor replication), Harbor skips artifacts that have already been successfully replicated. So, we should consider how much additional benefit this feature will bring.

Regarding that statement. there is a fundamental deficit with harbor replications, when the layer is in the copy process, new replication will try to copy the same blob again. If you have a slow connection like DSL the same blob will be copied over and over again. This results in the domino effect. Each new replication halves the throughput, resulting in slower transfers, and because all replications run in parallel, the same effect happens with the next blob/layer.

We have made an experiment with:

• Copying 1 GiB image
• Limit transfer speed 1 Mbit, per replication
  • Please note that we set the 1Mib per replication. On a real DSL, 3G connection, the total bandwidth (1 Mbit) would not be shared across all replications, so each replication would get 1/n of the bandwidth.
  • So the observed time needs to be multiplied by n.
• Replication every minute
• Replication time with "Prevent Overlapping Replications" Enabled ->: xxx min
• Replications time with "Prevent Overlapping Replications" Disabled -> : xxx min