Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4440

Enhance the job progress API to expose more information

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • Spark Core

    Description

      The progress API introduced in SPARK-2321 provides a new way for user to monitor job progress. However the information exposed in the API is relatively limited. It'll be much more useful if we can enhance the API to expose more data.
      Some improvement for example may include but not limited to:
      1. Stage submission and completion time.
      2. Task metrics.
      The requirement is initially identified for the hive on spark project(HIVE-7292), other application should benefit as well.

      Attachments

        Issue Links

          is depended upon by

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-3145 Hive on Spark umbrella

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-2321 Design a proper progress reporting & event listener API

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is required by

          Sub-task - The sub-task of the issue HIVE-9295 Cleanup code for getting spark job progress and metrics

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              Unassigned Unassigned
              lirui Rui Li
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: