Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5401

Executor ID should be set before MetricsSystem is created

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1, 1.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      MetricsSystem construction attempts to namespace metrics from each executor using that executor's ID.

      The ID is currently set at Executor construction time (uncoincidentally, just before the ExecutorSource is registered), but this is after the MetricsSystem has been initialized (which happens during SparkEnv construction, which itself happens during ExecutorBackend construction, before Executor construction).

      I noticed this problem because I wasn't seeing any JVM metrics from my executors in a Graphite dashboard I've set up; turns out all the executors (and the driver) were namespacing their metrics under "<driver>", and Graphite responds to such a situation by only taking the last value it receives for each "metric" within a configurable time window (e.g. 10s). I was seeing per-executor metrics, properly namespaced with each executor's ID, from ExecutorSource, which as I mentioned above is registered after the executor ID is set.

      I have a one-line fix for this that I will submit shortly.

        Attachments

          Activity

            People

            • Assignee:
              rdub Ryan Williams
              Reporter:
              rdub Ryan Williams
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: