Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2099

Report TaskMetrics for running tasks

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Spark currently collects a set of helpful task metrics, like shuffle bytes written, GC time, and displays them on the app web UI. These are only collected and displayed for tasks that have completed. This makes them unsuited to perhaps the situation where they would be most useful - determining what's going wrong in currently running tasks.

      Reporting metrics progrss for running tasks would probably require adding an executor->driver heartbeat that reports metrics for all tasks currently running on the executor.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sandyr Sandy Ryza
                Reporter:
                sandyr Sandy Ryza
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: