Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3620

Refactor config option handling code for spark-submit

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 1.0.0, 1.1.0
    • Fix Version/s: None
    • Component/s: Deploy
    • Labels:
      None

      Description

      I'm proposing its time to refactor the configuration argument handling code in spark-submit. The code has grown organically in a short period of time, handles a pretty complicated logic flow, and is now pretty fragile. Some issues that have been identified:

      1. Hand-crafted property file readers that do not support the property file format as specified in http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader)

      2. ResolveURI not called on paths read from conf/prop files

      3. inconsistent means of merging / overriding values from different sources (Some get overridden by file, others by manual settings of field on object, Some by properties)

      4. Argument validation should be done after combining config files, system properties and command line arguments,

      5. Alternate conf file location not handled in shell scripts

      6. Some options can only be passed as command line arguments

      7. Defaults for options are hard-coded (and sometimes overridden multiple times) in many through-out the code e.g. master = local[*]

      Initial proposal is to use typesafe conf to read in the config information and merge the various config sources

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tigerquoll Dale Richardson
                Reporter:
                tigerquoll Dale Richardson
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: