Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22190

Add Spark executor task metrics to Dropwizard metrics

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      I would like to propose to expose Spark executor task metrics using the Dropwizard metrics. I have developed a simple implementation and run a few tests using Graphite sink and Grafana visualization and this appears to me a good source of information for monitoring and troubleshooting the progress of Spark jobs. I attach a screenshot of an example graph generated with Grafana.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lucacanali Luca Canali
                Reporter:
                lucacanali Luca Canali
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: