Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Won't Fix
-
None
-
None
-
None
-
None
Description
Issues like OOZIE-2658 can happen with more properties.
| --conf | overwritten by |
|---|---|
| spark.driver.memory | --driver-memory |
| spark.driver.extraJavaOptions | --driver-java-options |
| spark.driver.extraLibraryPath | --driver-library-path |
| spark.executor.instances | --num-executors |
| spark.driver.extraClassPath | --driver-class-path |
| spark.yarn.dist.files | --files |
We should make sure that one is converted to the other (the current pattern is to use --conf versions of these) so we don't run into issues where the log4j properties are overwritten. (OOZIE-2705 can be affected by this)