Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11013

CLASSPATH handling should be consolidated, debuggable

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      As part of HADOOP-9902, java execution across many different shell bits were consolidated down to (effectively) two routines. Prior to calling those two routines, the CLASSPATH is exported. This export should really be getting handled in the exec function and not in the individual shell bits.

      Additionally, it would be good if there was:

      echo ${CLASSPATH} > /dev/null
      

      so that bash x would show the content of the classpath or even a '-debug classpath' option that would echo the classpath to the screen prior to java exec to help with debugging.

      1. HADOOP-11013.patch
        9 kB
        Allen Wittenauer
      2. HADOOP-11013-01.patch
        19 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment -

          I'm in the midst of rewriting slsrun.sh and it's very clear that this needs to happen first. It's only going to get worse with KMS and HTTPFS.

          Show
          aw Allen Wittenauer added a comment - I'm in the midst of rewriting slsrun.sh and it's very clear that this needs to happen first. It's only going to get worse with KMS and HTTPFS.
          Hide
          aw Allen Wittenauer added a comment -

          a) adds --debug option
          b) reworks CLASSPATH export to be consistent/safer

          Show
          aw Allen Wittenauer added a comment - a) adds --debug option b) reworks CLASSPATH export to be consistent/safer
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664808/HADOOP-11013.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4564//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664808/HADOOP-11013.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4564//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment - - edited

          Some sample output:

          $ bin/hdfs --debug --daemon start namenode
          DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
          DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
          DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
          DEBUG: HADOOP_OPTS declined -Xmx1024m
          DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log
          DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
          DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
          DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA
          DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
          DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
          
          aw-mbp-work:hadoop-3.0.0-SNAPSHOT aw$ more logs/*out
          DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
          DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=  -Djava.security.krb5.kdc=  -Djava.security.krb5.conf=  -Xmx1g -XX:HeapDumpPath=./java_pid_<pid>.hprof -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
          
          Show
          aw Allen Wittenauer added a comment - - edited Some sample output: $ bin/hdfs --debug --daemon start namenode DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/ native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/ native DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Added CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: HADOOP_OPTS declined -Xmx1024m DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender aw-mbp-work:hadoop-3.0.0-SNAPSHOT aw$ more logs/*out DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack= true -Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djava.security.krb5.conf= -Xmx1g -XX:HeapDumpPath=./java_pid_<pid>.hprof -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
          Hide
          cwimmer Charles Wimmer added a comment -

          This could have saved me hours of debugging classpath issues.

          Show
          cwimmer Charles Wimmer added a comment - This could have saved me hours of debugging classpath issues.
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM, thanks Allen

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM, thanks Allen
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • Replaced all of the if's with a function
          • Added more messages that fill in some of the blanks (e.g., when does HADOOP_NAMENODE_OPTS get added?)
          • Because I'm never happy, even with my own code
          Show
          aw Allen Wittenauer added a comment - -01: Replaced all of the if's with a function Added more messages that fill in some of the blanks (e.g., when does HADOOP_NAMENODE_OPTS get added?) Because I'm never happy, even with my own code
          Hide
          aw Allen Wittenauer added a comment -

          Some examples:

          Namenode:

          $ bin/hdfs --debug --daemon start namenode
          DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
          DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
          DEBUG: Appending HADOOP_NAMENODE_OPTS onto HADOOP_OPTS
          DEBUG: HADOOP_OPTS declined -Xmx1024m
          DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log
          DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
          DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
          DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA
          DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
          DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
          WARNING: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs does not exist. Creating.
          

          ResourceManager:

          $ bin/yarn --debug --daemon start resourcemanager
          DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
          DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
          DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
          DEBUG: Append YARN_RESOURCEMANAGER_OPTS onto YARN_OPTS
          DEBUG: Resetting HADOOP_OPTS=YARN_OPTS
          DEBUG: HADOOP_OPTS accepted -Xmx1024m
          DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
          DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log
          DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
          DEBUG: HADOOP_OPTS declined -Dyarn.root.logger=INFO,RFA
          DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log
          DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
          DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
          DEBUG: HADOOP_OPTS declined -Dhadoop.root.logger=INFO,RFA
          DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
          DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
          

          distcp

          $ bin/hadoop --debug distcp /tmp /1
          DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf
          DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/native
          DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/native
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*
          DEBUG: Injecting TOOL_PATH into CLASSPATH
          DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/*
          DEBUG: Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS
          DEBUG: HADOOP_OPTS accepted -Xmx1024m
          DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
          DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop.log
          DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
          DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw
          DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,console
          DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
          DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender
          DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/*
          DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm=  -Djava.security.krb5.kdc=  -Djava.security.krb5.conf=   -Xmx1024m -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
          
          Show
          aw Allen Wittenauer added a comment - Some examples: Namenode: $ bin/hdfs --debug --daemon start namenode DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/ native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/ native DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Appending HADOOP_NAMENODE_OPTS onto HADOOP_OPTS DEBUG: HADOOP_OPTS declined -Xmx1024m DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-namenode-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,RFA DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender WARNING: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs does not exist. Creating. ResourceManager: $ bin/yarn --debug --daemon start resourcemanager DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/ native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/ native DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Append YARN_RESOURCEMANAGER_OPTS onto YARN_OPTS DEBUG: Resetting HADOOP_OPTS=YARN_OPTS DEBUG: HADOOP_OPTS accepted -Xmx1024m DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS declined -Dyarn.root.logger=INFO,RFA DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS declined -Dhadoop.root.logger=INFO,RFA DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender distcp $ bin/hadoop --debug distcp /tmp /1 DEBUG: HADOOP_CONF_DIR=/Users/aw/HADOOP/conf DEBUG: Initial CLASSPATH=/Users/aw/HADOOP/conf DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/build/ native DEBUG: Rejected colonpath(JAVA_LIBRARY_PATH): /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/lib/ native DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/* DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/* DEBUG: Injecting TOOL_PATH into CLASSPATH DEBUG: Append CLASSPATH: /Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/* DEBUG: Appending HADOOP_CLIENT_OPTS onto HADOOP_OPTS DEBUG: HADOOP_OPTS accepted -Xmx1024m DEBUG: Dupe CLASSPATH: /Users/aw/HADOOP/conf DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop.log DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=aw DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,console DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml DEBUG: HADOOP_OPTS accepted -Dhadoop.security.logger=INFO,NullAppender DEBUG: Final CLASSPATH: /Users/aw/HADOOP/conf:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/*:/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/share/hadoop/tools/lib/* DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack= true -Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djava.security.krb5.conf= -Xmx1024m -Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
          Hide
          aw Allen Wittenauer added a comment -

          What's interesting to me in that output is the prevention of the dupe'ing of the HADOOP_CONF_DIR in the CLASSPATH. It could be (easily) argued that HADOOP_CONF_DIR should only be added once, but I'm leaning towards leaving it in there like it currently is as a "safety" feature.

          Show
          aw Allen Wittenauer added a comment - What's interesting to me in that output is the prevention of the dupe'ing of the HADOOP_CONF_DIR in the CLASSPATH. It could be (easily) argued that HADOOP_CONF_DIR should only be added once, but I'm leaning towards leaving it in there like it currently is as a "safety" feature.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664993/HADOOP-11013-01.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4568//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664993/HADOOP-11013-01.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4568//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Committing this to trunk, given the differences between the patches is minor.

          Show
          aw Allen Wittenauer added a comment - Committing this to trunk, given the differences between the patches is minor.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #663 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/663/)
          HADOOP-11013. CLASSPATH handling should be consolidated, debuggable (aw) (aw: rev d8774cc577198fdc3bc36c26526c95ea9a989800)

          • hadoop-common-project/hadoop-common/src/main/bin/rcc
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #663 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/663/ ) HADOOP-11013 . CLASSPATH handling should be consolidated, debuggable (aw) (aw: rev d8774cc577198fdc3bc36c26526c95ea9a989800) hadoop-common-project/hadoop-common/src/main/bin/rcc hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-mapreduce-project/bin/mapred hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1854 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1854/)
          HADOOP-11013. CLASSPATH handling should be consolidated, debuggable (aw) (aw: rev d8774cc577198fdc3bc36c26526c95ea9a989800)

          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/rcc
          • hadoop-mapreduce-project/bin/mapred
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1854 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1854/ ) HADOOP-11013 . CLASSPATH handling should be consolidated, debuggable (aw) (aw: rev d8774cc577198fdc3bc36c26526c95ea9a989800) hadoop-yarn-project/hadoop-yarn/bin/yarn hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/rcc hadoop-mapreduce-project/bin/mapred
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1880 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1880/)
          HADOOP-11013. CLASSPATH handling should be consolidated, debuggable (aw) (aw: rev d8774cc577198fdc3bc36c26526c95ea9a989800)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/rcc
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-mapreduce-project/bin/mapred
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/bin/yarn
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1880 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1880/ ) HADOOP-11013 . CLASSPATH handling should be consolidated, debuggable (aw) (aw: rev d8774cc577198fdc3bc36c26526c95ea9a989800) hadoop-common-project/hadoop-common/src/main/bin/hadoop hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/rcc hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-mapreduce-project/bin/mapred hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/bin/yarn

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development