add better support for integrating with external cluster manager

[dejanzele]

What changes were proposed in this pull request?

• Remove previous checks in SparkSubmit that restricted use of external managers to specific master URLs.
• Add new configuration field spark.submit.external.class.

Why are the changes needed?

SparkSubmit currently only supports built-in cluster managers (YARN, Kubernetes, Standalone), making it difficult to integrate custom or third-party schedulers without forking the codebase.
By removing hard-coded restrictions, this patch enables easier integration with various cluster managers.

Now the following is possible (assuming the implementation is loaded):

bin/spark-submit                    \          
  --master armada://localhost:30002 \
  --deploy-mode cluster 

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Added unit tests and tested with armadaproject/armada-spark custom integration.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

Can we file a JIRA? See also https://spark.apache.org/contributing.html