Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new configuration parameters heuristic #2

Open
wants to merge 1 commit into
base: stage_analysis
Choose a base branch
from

Conversation

edwinalu
Copy link
Owner

Add new configuration parameters heuristic, which will list the current values for configuration parameters, and also recommended new values. To determine new values, it will check for:

  • execution memory spill: this will slow down the application, so try to prevent this by increasing partitions, increasing executor memory, or decreasing cores.
  • long tasks: this will slow down the application, so try to prevent this by increasing the number of partitions.
  • task skew: this will slow down the application, so add recommendations for making partitions more even.
  • OOM or GC: increase memory, increase partitions, or decrease cores, to try to avoid the error.
  • container killed by YARN errors: increase overhead memory.
  • unused executor memory, if this is much higher than max JVM used memory; either increase cores or decrease memory.
  • driver configuration parameters (memory and cores).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant