Hadoop Common
  1. Hadoop Common
  2. HADOOP-10623

Provide a utility to be able inspect the config as seen by a hadoop client / daemon

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      To ease debugging of config issues it is convenient to be able to generate a config as seen by the job client or a hadoop daemon

      ]$ hadoop org.apache.hadoop.util.ConfigTool -help 
      Usage: ConfigTool [ -xml | -json ] [ -loadDefaults ] [ resource1... ]
            if resource contains '/', load from local filesystem
            otherwise, load from the classpath
      
      Generic options supported are
      -conf <configuration file>     specify an application configuration file
      -D <property=value>            use value for given property
      -fs <local|namenode:port>      specify a namenode
      -jt <local|jobtracker:port>    specify a job tracker
      -files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
      -libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
      -archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.
      
      The general command line syntax is
      bin/hadoop command [genericOptions] [commandOptions]
      
      $ hadoop org.apache.hadoop.util.ConfigTool -Dmy.test.conf=val mapred-site.xml ./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml | python -mjson.tool
      {
          "properties": [
              {
                  "isFinal": false,
                  "key": "mapreduce.framework.name",
                  "resource": "mapred-site.xml",
                  "value": "yarn"
              },
              {
                  "isFinal": false,
                  "key": "mapreduce.client.genericoptionsparser.used",
                  "resource": "programatically",
                  "value": "true"
              },
              {
                  "isFinal": false,
                  "key": "my.test.conf",
                  "resource": "from command line",
                  "value": "val"
              },
              {
                  "isFinal": false,
                  "key": "from.file.key",
                  "resource": "hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml",
                  "value": "from.file.val"
              },
              {
                  "isFinal": false,
                  "key": "mapreduce.shuffle.port",
                  "resource": "mapred-site.xml",
                  "value": "${my.mapreduce.shuffle.port}"
              }
          ]
      }
      
      1. HADOOP-10623.v01.patch
        4 kB
        Gera Shegalov
      2. HADOOP-10623.v02.patch
        6 kB
        Gera Shegalov
      3. HADOOP-10623.v03.patch
        7 kB
        Gera Shegalov
      4. HADOOP-10623.v04.patch
        7 kB
        Gera Shegalov

        Issue Links

          Activity

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development