Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Consider core-seconds in TCL heuristics #1307

Closed
2 tasks done
cindyyuanjiang opened this issue Aug 21, 2024 · 2 comments
Closed
2 tasks done

[FEA] Consider core-seconds in TCL heuristics #1307

cindyyuanjiang opened this issue Aug 21, 2024 · 2 comments
Assignees
Labels
feature request New feature or request user_tools Scope the wrapper module running CSP, QualX, and reports (python)

Comments

@cindyyuanjiang
Copy link
Collaborator

cindyyuanjiang commented Aug 21, 2024

Is your feature request related to a problem? Please describe.
We found a job with low compute and ran on a cheaper instance type on Databricks does not run faster with GPU. We should factor in minimum core second threshold in Top Candidate View.

@cindyyuanjiang cindyyuanjiang self-assigned this Aug 21, 2024
@cindyyuanjiang cindyyuanjiang added user_tools Scope the wrapper module running CSP, QualX, and reports (python) and removed ? - Needs Triage labels Aug 21, 2024
@mattahrens
Copy link
Collaborator

Initial scope is to calculate core seconds and add it as a metric in summary output CSV file and possibly CLI output as well.

Calculation for core seconds is the total executor time (by finding start/end time for each executor to get each executor duration) multiplied by the spark.executor.cores value for the workload.

@amahussein amahussein changed the title [FEA] Consider Core Seconds in Top Candidate View Heuristics [FEA] Consider core-seconds in TCL heuristics Aug 28, 2024
@cindyyuanjiang
Copy link
Collaborator Author

Closing as all subtasks are done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Development

No branches or pull requests

2 participants