Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5778

Throw if nonexistent "spark.metrics.conf" file is provided

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.2.1
    • 1.3.0
    • Spark Core
    • None

    Description

      Spark looks for a MetricsSystem configuration file when the spark.metrics.conf parameter is set, defaulting to the path "metrics.properties" when it's not set.

      In the event of a failure to find or parse the file, the exception is caught and an error is logged.

      This seems like reasonable behavior in the general case where the user has not specified a spark.metrics.conf file. However, I've been bitten several times by having specified a file that all or some executors did not have present (I typo'd the path, or forgot to add an additional --files flag to make my local metrics config file get shipped to all executors), the error was swallowed and I was confused about why I'd captured no metrics from a job that appeared to have run successfully.

      I'd like to change the behavior to actually throw if the user has specified a configuration file that doesn't exist.

      Attachments

        Activity

          People

            rdub Ryan Williams
            rdub Ryan Williams
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: