Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35502

Spark Executor metrics are not produced/showed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.0.1
    • None
    • Spark Submit
    • None

    Description

      Recently we have enabled prometheusServlet configuration in order to have spark master, worker, driver and executor metrics.

      We can see and using spark master, worker and driver executors but can't see spark executor metrics.

      We are running spark streaming standalone cluster in version 3.0.1 over physical servers.

       
      We have taken one of our jobs and added the following parameters to the job configuration, but couldn't see executer metrics by curling both driver and executor workers of this job:
       
      These are the parameters:
      --conf spark.ui.prometheus.enabled=true \
      --conf spark.executor.processTreeMetrics.enabled=true
       
      Curl commands:
      [00764f](root@sparktest-40005-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
      [00764f](root@sparktest-40005-prod-chidc2:~)#
      Driver of this job - sparktest-40004:
      [e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
      [e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
       
      Our UI port is on 4050
       
       I understood that the executor Prometheus endpoint is still experimental which may explain the inconsistent behaviour we see but is there a plan to fix it?
       
      Are there any known issues regarding this?
       
       
       
       
       

      Attachments

        Activity

          People

            Unassigned Unassigned
            mmizrahi Mati
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: