[SPARK-7169] Allow to specify metrics configuration more flexibly

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.2, 1.3.1
• Fix Version/s: 1.5.0
• Component/s: Spark Core
• Target Version/s: 1.4.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Jacek Lewandowski
• Created: 27/Apr/15 15:23
• Updated: 13/Jun/18 22:15
• Resolved: 07/Jun/15 04:08

Description

Metrics are configured in metrics.properties file. Path to this file is specified in SparkConf at a key spark.metrics.conf. The property is read when MetricsSystem is created which means, during SparkEnv initialisation.

Problem

When the user runs his application he has no way to provide the metrics configuration for executors. Although one can specify the path to metrics configuration file (1) the path is common for all the nodes and the client machine so there is implicit assumption that all the machines has same file in the same location, and (2) actually the user needs to copy the file manually to the worker nodes because the file is read before the user files are populated to the executor local directories. All of this makes it very difficult to play with the metrics configuration.

Proposed solution

I think that the easiest and the most consistent solution would be to move the configuration from a separate file directly to SparkConf. We may prefix all the configuration settings from the metrics configuration by, say spark.metrics.props. For the backward compatibility, these properties would be loaded from the specified as it works now. Such a solution doesn't change the API so maybe it could be even included in patch release of Spark 1.2 and Spark 1.3.

Appreciate any feedback.

Issue Links

• is related to: SPARK-5152 Let metrics.properties file take an hdfs:// path
• links to: [Github] Pull Request #5788 (jacek-lewandowski)
• links to: [Github] Pull Request #6560 (vanzin)