Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11828

DAGScheduler source registered too early with MetricsSystem

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      I see this log message when starting apps on YARN:

      15/11/18 13:12:56 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

      That's because DAGScheduler registers itself with the metrics system in its constructor, and the DAGScheduler is instantiated before "spark.app.id" is set in the context's SparkConf.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: