Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.2.1
-
None
Description
Spark looks for a MetricsSystem configuration file when the spark.metrics.conf parameter is set, defaulting to the path "metrics.properties" when it's not set.
In the event of a failure to find or parse the file, the exception is caught and an error is logged.
This seems like reasonable behavior in the general case where the user has not specified a spark.metrics.conf file. However, I've been bitten several times by having specified a file that all or some executors did not have present (I typo'd the path, or forgot to add an additional --files flag to make my local metrics config file get shipped to all executors), the error was swallowed and I was confused about why I'd captured no metrics from a job that appeared to have run successfully.
I'd like to change the behavior to actually throw if the user has specified a configuration file that doesn't exist.
