Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35502

Spark Executor metrics are not produced/showed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.1
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None
    • Environment:

      Description

      Recently we have enabled prometheusServlet configuration in order to have spark master, worker, driver and executor metrics.

      We can see and using spark master, worker and driver executors but can't see spark executor metrics.

      We are running spark streaming standalone cluster in version 3.0.1 over physical servers.

       
      We have taken one of our jobs and added the following parameters to the job configuration, but couldn't see executer metrics by curling both driver and executor workers of this job:
       
      These are the parameters:
      --conf spark.ui.prometheus.enabled=true \
      --conf spark.executor.processTreeMetrics.enabled=true
       
      Curl commands:
      [00764f](root@sparktest-40005-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
      [00764f](root@sparktest-40005-prod-chidc2:~)#
      Driver of this job - sparktest-40004:
      [e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
      [e35005](root@sparktest-40004-prod-chidc2:~)# curl -s [http://localhost:4050/metrics/executors/prometheus|head|http://localhost:4050/metrics/executors/prometheus%7Chead] -n5
       
      Our UI port is on 4050
       
       I understood that the executor Prometheus endpoint is still experimental which may explain the inconsistent behaviour we see but is there a plan to fix it?
       
      Are there any known issues regarding this?
       
       
       
       
       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mmizrahi Mati
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: