Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22343

Add support for publishing Spark metrics into Prometheus

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:

      Description

      I've created a PR (https://github.com/apache-spark-on-k8s/spark/pull/531) to supporting publishing Spark metrics into Prometheus metrics in the https://github.com/apache-spark-on-k8s/spark fork (Spark on Kubernetes). According to the maintainers of the project I should create a ticket here as well, in order to be tracked upstream. See below the original text of the PR:
      _
      Publishing Spark metrics into Prometheus - as discussed earlier in https://github.com/apache-spark-on-k8s/spark/pull/384.

      Implemented a metrics sink that publishes Spark metrics into Prometheus via [Prometheus Pushgateway](https://prometheus.io/docs/instrumenting/pushing/). Metrics data published by Spark is based on [Dropwizard](http://metrics.dropwizard.io/). The format of Spark metrics is not supported natively by Prometheus thus these are converted using [DropwizardExports](https://prometheus.io/client_java/io/prometheus/client/dropwizard/DropwizardExports.html) prior pushing metrics to the pushgateway.

      Also the default Prometheus pushgateway client API implementation does not support metrics timestamp thus the client API has been ehanced to enrich metrics data with timestamp. _

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              matyix_ Janos Matyas
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: