Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial changes for adding Spark stage analysis #438

Open
wants to merge 8 commits into
base: customSHSWork
Choose a base branch
from
Open

Initial changes for adding Spark stage analysis #438

wants to merge 8 commits into from

Commits on Oct 24, 2018

  1. Initial changes for adding Spark stage analysis. 

    - Add StageAnalyzer for analyzing the stages for a Spark application for execution memory spill,
  long tasks, task skew, and failures.
- Call REST API for getting failed tasks.
- Modify call to stages REST API to get task and executor summaries.
    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    dca51b8 View commit details
    Browse the repository at this point in the history

  2. Add keys for thresholds, comments, getDuration test.

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    aafd7d4 View commit details
    Browse the repository at this point in the history

  3. factor out results for different heuristics

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    2f6adbf View commit details
    Browse the repository at this point in the history

  4. code review comments

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    fbd4e79 View commit details
    Browse the repository at this point in the history

  5. lower DEFAULT_MAX_DATA_PROCESSED_THRESHOLD

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    af09da0 View commit details
    Browse the repository at this point in the history

  6. move shared values to StageAnalysisResult

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    c5d421a View commit details
    Browse the repository at this point in the history

  7. check fetchFailedTasks parameter

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    7966c61 View commit details
    Browse the repository at this point in the history

  8. code review comments

    @edwinalu
    edwinalu committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    135bb05 View commit details
    Browse the repository at this point in the history