Fixes #1476.

With the merge of arguments for toggling distributed mode in the Qualification Tool CLI, the next step is to implement the functionality for this feature.

Usage

spark_rapids qualification --platform onprem --eventlogs /path/to/eventlogs  --verbose --filter_apps all \
 --submission_mode distributed --tools_config_file /path/to/custom_conf_file.yaml

Notes:

• Sample config file - https://github.com/NVIDIA/spark-rapids-tools/blob/spark-rapids-tools-distributed-base/user_tools/tests/spark_rapids_tools_ut/resources/tools_config/valid/tools_config_01.yaml
• An document with setup steps has been shared internally. Feel free to reach out to me for any questions related to the setup.

Detailed Changes

Support for distributed job submission:

• user_tools/src/spark_rapids_pytools/rapids/rapids_job.py: Implemented the _submit_job method to support submitting the Tools JAR to a Spark cluster using DistributedToolsExecutor.
• user_tools/src/spark_rapids_tools_distributed/distributed_main.py: Added a new module to orchestrate the execution of the Spark Rapids Tools JAR on Spark, including environment validation, job submission, and output processing.

Enhancements to utility functions:

• user_tools/src/spark_rapids_tools/utils/util.py: Added parse_memory_size_in_gb and archive_directory methods to convert memory size strings to gigabytes and archive directories, respectively.
• user_tools/src/spark_rapids_tools/storagelib/csppath.py: Added open_append_stream method to support appending to files.

New dependencies:

• user_tools/pyproject.toml: Added pyspark dependency for distributed computing.

Other notable changes:

• user_tools/src/spark_rapids_pytools/rapids/qualification.py: Added a check to log a warning and return an empty DataFrame if the report file does not exist.
• user_tools/src/spark_rapids_tools_distributed/jar_cmd_args.py: Added cached properties to extract specific JVM arguments and paths from the command arguments.