Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-345

JobConf access to name-values

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • None

    Description

      class JobConf (or its base class Configuration)
      should be extended to enable enumeration of all its key-value pairs.
      ( more precisely: the Properties returned by Configuration.getProps() )

      This will be useful to "export" all JobConf properties to environment variables.
      We use env.vars to expose some Hadoop context to non-Java MapReduce applications.

      Note that the typed properties are also represented as Strings
      (getInt, getStrings, getClass, etc.)
      So a single enumeration exposes everything as (untyped) environment variables.

      The proposed escaping rules from JobConf properties to env.var are:

      1. values are left as-is.
      2. keys are escaped as follows:
      [A-Za-z0-9] --> unchanged.
      all other chars --> underscore.

      For example
      set("mapred.input.key.class", "com.example.MyKey")
      becomes env.var:
      export mapred_input_key_class=com.example.MyKey

      Justification:
      1. Environment variables are case-sensitive. (Although uppercase is the preferred convention)
      So no need to uppercase everything.
      2. Some characters are forbidden in env.vars, or at least not shell-friendly:
      For example period, colon are problematic.
      3. The Hadoop conventions are already hierarchical and provide some namespace protection.
      This means we don't need an additional prefix as protection.
      For example all exported environment variables will start with "mapred." , "dfs." , "ipc." etc.
      This means they will not conflict with standard environemnt variables like PATH, USER, etc.
      And they will not conflict with standard hadoop env.vars because those are upper-case. (like HADOOP_CONF_DIR)

      Attachments

        1. config.entries.patch
          0.6 kB
          Michel Tourn

        Activity

          People

            michel_tourn Michel Tourn
            michel_tourn Michel Tourn
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: