Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.3.0
-
None
Description
Currently, code in SparkSubmit cannot call classes or methods that initialize the Spark Logging framework. That is because at that time SparkSubmit doesn't yet know which application will run, and logging is initialized differently for certain special applications (notably, the shells).
It would be better if either SparkSubmit did logging initialization earlier based on the application to be run, or did it in a way that could be overridden later when the app initializes.
Without this, there are currently a few parts of SparkSubmit that duplicates code from other parts of Spark just to avoid logging. For example:
- downloadFiles replicates code from Utils.scala
- createTempDir replicates code from Utils.scala and installs its own shutdown hook
- a few parts of the code could use SparkConf but can't right now because of the logging issue.
