Update Databricks Tools Notebooks for Tools v24.06.0

[parthosa]

Fixes #380. This PR fixes the databricks notebook for compatibility with Tools v24.06.0. Would recommend feedback by testing the changes E2E.

Changes

Qualification User Tools Notebook Template:

1. Fix virtual environment setup for user tools notebook.
   • Using %pip install spark_rapids_user_tools directly since the libraries are automatically notebook scoped and we do not need to create an additional virtual environment.
2. Add Tools Version argument to use custom tools version.
3. Show Console Output as well since it contains the Top Candidates view
4. Fix summary section - Currently it shows rapids_4_spark_qualification_output.csv but for user tools it should be qualification_summary.csv.
5. Include a Download button that downloads a zip file with output files and logs.

Future Maintenence:

1. Added a line in the README to record the latest version of tools used to test the notebooks.
2. For future, If we want to make newer version of tools as default:
   • Test the notebooks for this newer version.
   • If everything looks good, we should update the version number in this line.
   • Update TOOLS_VER variable in each notebook with the new version number.

Evaluation

1. Tested for Databricks AWS and Azure platforms.
2. Tested for the following file locations:
   • /dbfs/<eventlog>
   • Cannot support s3 and abfs locations because aws and az CLI are not available.

cc: @amahussein

[SurajAralihalli]

Cannot support s3 and abfs locations because aws and az CLI are not available.
Note User Tools Notebook might still fail since aws and az CLI cmds are not available.

We can include these limitations in the readme.

[cindyyuanjiang]

We should consider moving output info to cells after running the qualification tool. It would be nice if the first cell contains a concise summary of what the qual tool does.

[parthosa]

Address review feedback and included a download button to download output results as a zip along with log files.

[nvliyuan]

Hi @parthosa , can we target this pr to branch-24.06? This repo follows the same release strategy as the plugin, thx.

[parthosa]

@nvliyuan, Updated the target PR to branch-24.06. Thank you.

[SurajAralihalli]

LTGM, tested the notebooks on Azure Databricks.

[cindyyuanjiang]

LGTM, tested both notebooks. Thanks @parthosa!

Thank you for the reviews. Holding it off till next minor release of tools which handles the error when CSP CLIs are not present (in this case databricks and aws) NVIDIA/spark-rapids-tools#1035

[parthosa]

Based on offline discussion with @viadea:

• Added a note with the current limitation of using single event logs for the Qualification notebook.
• We can merge this PR for now.
• Plan to submit a subsequent PR to update the notebooks to use newer version of tools.

Result

[kuhushukla]

@parthosa what related issue with respect to single log (csp cli creds) needs to be fixed first in order for that limitation to resolve? I see that NVIDIA/spark-rapids-tools#1035 is merged.

[parthosa]

Yes @kuhushukla, it is merged and will be available to users from next release.

[kuhushukla]

Yes @kuhushukla, it is merged and will be available to users from next release.

In that case, using this for multiple logs note would be deprecated even as part of this change right?

[kuhushukla]

lgtm