From 765d48d0d621538a2d713bb8e5736864cedc89bc Mon Sep 17 00:00:00 2001 From: Demogorgon314 Date: Fri, 13 Sep 2024 15:11:17 +0800 Subject: [PATCH 1/2] PIP-379: Support setting up specific namespaces to skipping the load-shedding --- pip/pip_379.md | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 pip/pip_379.md diff --git a/pip/pip_379.md b/pip/pip_379.md new file mode 100644 index 0000000000000..df08999c2dafa --- /dev/null +++ b/pip/pip_379.md @@ -0,0 +1,80 @@ +# PIP-379: Support setting up specific namespaces to skipping the load-shedding + +# Background knowledge + +Some of the Pulsar protocols, for example the KoP will use a system topic to store the metadata, eg: the topic's offset, +when the bundle owner changes the topic's data will need to be full recovered to the broker, and when unloaded, +it needs to remove the broker local memory, if the topic's data were huge, it may cause slow loading. + +# Motivation + +Support setting up specific namespaces to skipping the load-shedding, for the system topics namespace, +or for benchmark-relevant namespaces we better be skipping the load shedding because for system topics, +if the ownership transfers to another broker, it may need to recover the data by replaying the messages, +it will cause message pub/sub temporary unavailable. + +# Goals + +## In Scope + +- Support setting up specific namespaces to skipping the load-shedding +- Use `RoundRobinBrokerSelector` to assign the skipped namespaces bundle. + +# High Level Design + +Add new configuration `loadBalancerSheddingExcludedNamespaces` to the `ServiceConfiguration`. +While doing the load-shedding, filter out the configuration namespaces. + +For shedding excluded namespaces, use `RoundRobinBrokerSelector` to assign the ownership, it can make the assignment +more average because these will not automatically rebalance to another broker unless manually unloaded it. + +# Detailed Design + +## Design & Implementation Details + +## Public-facing Changes + +### Configuration + +Add new configuration `loadBalancerSheddingExcludedNamespaces` to the `ServiceConfiguration`. + +```java + @FieldContext( + dynamic = true, + category = CATEGORY_LOAD_BALANCER, + doc = "The namespaces skip for load shedding" + ) + private Set loadBalancerSheddingExcludedNamespaces = new TreeSet<>();; +``` + +# Monitoring + +No new metrics are added in this proposal. + +# Security Considerations + +No new security considerations are added in this proposal. + +# Backward & Forward Compatibility + +## Revert + +No changes are needed to revert to the previous version. + +## Upgrade + +No other changes are needed to upgrade to the new version. + +# Alternatives + +None + +# General Notes + +# Links + + +* Mailing List discussion thread: +* Mailing List voting thread: From 59f4f861c2053a75c2f5e3b9a08ca59510869f54 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Tue, 17 Sep 2024 21:45:48 -0700 Subject: [PATCH 2/2] rename pip_379.md to pip_380.md --- pip/{pip_379.md => pip_380.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename pip/{pip_379.md => pip_380.md} (97%) diff --git a/pip/pip_379.md b/pip/pip_380.md similarity index 97% rename from pip/pip_379.md rename to pip/pip_380.md index df08999c2dafa..982f8da846555 100644 --- a/pip/pip_379.md +++ b/pip/pip_380.md @@ -1,4 +1,4 @@ -# PIP-379: Support setting up specific namespaces to skipping the load-shedding +# PIP-380: Support setting up specific namespaces to skipping the load-shedding # Background knowledge