Skip to content

Commit 94a043d

Browse files
pan3793HyukjinKwon
authored andcommitted
[SPARK-45670][CORE][3.3] SparkSubmit does not support --total-executor-cores when deploying on K8s
This is the cherry-pick of #43536 for branch-3.3 ### What changes were proposed in this pull request? Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit ### Why are the changes needed? `--total-executor-cores` does not take effect in Spark on K8s, [the comments from original PR](#19717 (comment)) also proves that ### Does this PR introduce _any_ user-facing change? The output of `spark-submit --help` changed ```patch ... - Spark standalone, Mesos and Kubernetes only: + Spark standalone and Mesos only: --total-executor-cores NUM Total cores for all executors. ... ``` ### How was this patch tested? Pass GA and review. ### Was this patch authored or co-authored using generative AI tooling? No Closes #43548 from pan3793/SPARK-45670-3.3. Authored-by: Cheng Pan <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 1bb4143 commit 94a043d

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

+1-1
Original file line numberDiff line numberDiff line change
@@ -647,7 +647,7 @@ private[spark] class SparkSubmit extends Logging {
647647
confKey = EXECUTOR_CORES.key),
648648
OptionAssigner(args.executorMemory, STANDALONE | MESOS | YARN | KUBERNETES, ALL_DEPLOY_MODES,
649649
confKey = EXECUTOR_MEMORY.key),
650-
OptionAssigner(args.totalExecutorCores, STANDALONE | MESOS | KUBERNETES, ALL_DEPLOY_MODES,
650+
OptionAssigner(args.totalExecutorCores, STANDALONE | MESOS, ALL_DEPLOY_MODES,
651651
confKey = CORES_MAX.key),
652652
OptionAssigner(args.files, LOCAL | STANDALONE | MESOS | KUBERNETES, ALL_DEPLOY_MODES,
653653
confKey = FILES.key),

core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

+1-1
Original file line numberDiff line numberDiff line change
@@ -550,7 +550,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
550550
| --kill SUBMISSION_ID If given, kills the driver specified.
551551
| --status SUBMISSION_ID If given, requests the status of the driver specified.
552552
|
553-
| Spark standalone, Mesos and Kubernetes only:
553+
| Spark standalone and Mesos only:
554554
| --total-executor-cores NUM Total cores for all executors.
555555
|
556556
| Spark standalone, YARN and Kubernetes only:

0 commit comments

Comments
 (0)