Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10623

Provide a utility to be able inspect the config as seen by a hadoop client / daemon

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      To ease debugging of config issues it is convenient to be able to generate a config as seen by the job client or a hadoop daemon

      ]$ hadoop org.apache.hadoop.util.ConfigTool -help 
      Usage: ConfigTool [ -xml | -json ] [ -loadDefaults ] [ resource1... ]
            if resource contains '/', load from local filesystem
            otherwise, load from the classpath
      
      Generic options supported are
      -conf <configuration file>     specify an application configuration file
      -D <property=value>            use value for given property
      -fs <local|namenode:port>      specify a namenode
      -jt <local|jobtracker:port>    specify a job tracker
      -files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
      -libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
      -archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.
      
      The general command line syntax is
      bin/hadoop command [genericOptions] [commandOptions]
      
      $ hadoop org.apache.hadoop.util.ConfigTool -Dmy.test.conf=val mapred-site.xml ./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml | python -mjson.tool
      {
          "properties": [
              {
                  "isFinal": false,
                  "key": "mapreduce.framework.name",
                  "resource": "mapred-site.xml",
                  "value": "yarn"
              },
              {
                  "isFinal": false,
                  "key": "mapreduce.client.genericoptionsparser.used",
                  "resource": "programatically",
                  "value": "true"
              },
              {
                  "isFinal": false,
                  "key": "my.test.conf",
                  "resource": "from command line",
                  "value": "val"
              },
              {
                  "isFinal": false,
                  "key": "from.file.key",
                  "resource": "hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml",
                  "value": "from.file.val"
              },
              {
                  "isFinal": false,
                  "key": "mapreduce.shuffle.port",
                  "resource": "mapred-site.xml",
                  "value": "${my.mapreduce.shuffle.port}"
              }
          ]
      }
      

      Attachments

        1. HADOOP-10623.v01.patch
          4 kB
          Gera Shegalov
        2. HADOOP-10623.v02.patch
          6 kB
          Gera Shegalov
        3. HADOOP-10623.v03.patch
          7 kB
          Gera Shegalov
        4. HADOOP-10623.v04.patch
          7 kB
          Gera Shegalov

        Issue Links

          Activity

            People

              jira.shegalov Gera Shegalov
              jira.shegalov Gera Shegalov
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: