[HADOOP-10623] Provide a utility to be able inspect the config as seen by a hadoop client / daemon

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Gera Shegalov
• Reporter: Gera Shegalov
• Created: 21/May/14 03:55
• Updated: 07/Jan/17 01:57

Description

To ease debugging of config issues it is convenient to be able to generate a config as seen by the job client or a hadoop daemon

]$ hadoop org.apache.hadoop.util.ConfigTool -help 
Usage: ConfigTool [ -xml | -json ] [ -loadDefaults ] [ resource1... ]
      if resource contains '/', load from local filesystem
      otherwise, load from the classpath

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|jobtracker:port>    specify a job tracker
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

$ hadoop org.apache.hadoop.util.ConfigTool -Dmy.test.conf=val mapred-site.xml ./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml | python -mjson.tool
{
    "properties": [
        {
            "isFinal": false,
            "key": "mapreduce.framework.name",
            "resource": "mapred-site.xml",
            "value": "yarn"
        },
        {
            "isFinal": false,
            "key": "mapreduce.client.genericoptionsparser.used",
            "resource": "programatically",
            "value": "true"
        },
        {
            "isFinal": false,
            "key": "my.test.conf",
            "resource": "from command line",
            "value": "val"
        },
        {
            "isFinal": false,
            "key": "from.file.key",
            "resource": "hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml",
            "value": "from.file.val"
        },
        {
            "isFinal": false,
            "key": "mapreduce.shuffle.port",
            "resource": "mapred-site.xml",
            "value": "${my.mapreduce.shuffle.port}"
        }
    ]
}

Issue Links

• is related to: HADOOP-7947 Validate XMLs if a relevant tool is available, when using scripts
• is related to: HADOOP-9044 add FindClass main class to provide classpath checking of installations
• is related to: HADOOP-12118 Validate xml configuration files with XML Schema