Hadoop Common
  1. Hadoop Common
  2. HADOOP-9005

Merge hadoop cmd line scripts from branch-1-win

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: trunk-win
    • Fix Version/s: trunk-win
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Tracking Jira for merging hadoop cmd line scripts from branch-1-win to trunk. Scripts also have to be updated to reflect their unix equivalents.

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          Missed other contributors to this patch. Thank you David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.

          Show
          Suresh Srinivas added a comment - Missed other contributors to this patch. Thank you David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.
          Hide
          Suresh Srinivas added a comment -

          +1

          Thank you Ivan. I committed the patch to branch-trunk-win.

          Show
          Suresh Srinivas added a comment - +1 Thank you Ivan. I committed the patch to branch-trunk-win.
          Hide
          Chris Nauroth added a comment -

          +1 for the latest patch

          I tested successfully on my Windows VM. Thank you for taking care of the assembly definitions too.

          Show
          Chris Nauroth added a comment - +1 for the latest patch I tested successfully on my Windows VM. Thank you for taking care of the assembly definitions too.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12552361/HADOOP-9005.branch-trunk-win.scripts.7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12552361/HADOOP-9005.branch-trunk-win.scripts.7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//console This message is automatically generated.
          Hide
          Ivan Mitic added a comment -

          Attaching updated patch.

          The patch also includes basic start/stop scripts so that we can unblock getting trunk running on Windows. Start/stop scripts are not complete at this point, that will be addressed thru a separate Jira.

          Show
          Ivan Mitic added a comment - Attaching updated patch. The patch also includes basic start/stop scripts so that we can unblock getting trunk running on Windows. Start/stop scripts are not complete at this point, that will be addressed thru a separate Jira.
          Hide
          Chris Nauroth added a comment -

          2. Please disregard this comment. My original comment was in reference to hadoop.cmd (not hadoop-env.cmd), but I see now that this is equivalent to the sh version anyway. Thank you.

          Show
          Chris Nauroth added a comment - 2. Please disregard this comment. My original comment was in reference to hadoop.cmd (not hadoop-env.cmd), but I see now that this is equivalent to the sh version anyway. Thank you.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//console This message is automatically generated.
          Hide
          Ivan Mitic added a comment -

          Thanks for clarifying Chris!

          2. Hmm, I still don't see the problem.
          This is the snippet from the shell script:

          # Command specific options appended to HADOOP_OPTS when specified
          export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
          export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
          export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
          

          And this is the cmd script equivalent:

          @rem Command specific options appended to HADOOP_OPTS when specified
          if not defined HADOOP_SECURITY_LOGGER (
            set HADOOP_SECURITY_LOGGER=INFO,RFAS
          )
          if not defined HDFS_AUDIT_LOGGER (
            set HDFS_AUDIT_LOGGER=INFO,NullAppender
          )
          
          set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS%
          set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS%
          set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS%
          

          3. Sure, I'll fix this.

          Show
          Ivan Mitic added a comment - Thanks for clarifying Chris! 2. Hmm, I still don't see the problem. This is the snippet from the shell script: # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_DATANODE_OPTS= "-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" And this is the cmd script equivalent: @rem Command specific options appended to HADOOP_OPTS when specified if not defined HADOOP_SECURITY_LOGGER ( set HADOOP_SECURITY_LOGGER=INFO,RFAS ) if not defined HDFS_AUDIT_LOGGER ( set HDFS_AUDIT_LOGGER=INFO,NullAppender ) set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS% set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS% set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS% 3. Sure, I'll fix this.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          2. I see lines like this in hadoop-env.sh. It seems audit logger is disabled instead of security logger.

          export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
          export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
          

          3. Most of the variables look to be set up with commented out lines for people to uncomment and use. i.e.:

          @rem Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
          @rem set HADOOP_CLASSPATH=
          

          I noticed that there is no equivalent "@rem set JAVA_HOME" line.

          Show
          Chris Nauroth added a comment - 2. I see lines like this in hadoop-env.sh. It seems audit logger is disabled instead of security logger. export HADOOP_NAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" 3. Most of the variables look to be set up with commented out lines for people to uncomment and use. i.e.: @rem Extra Java CLASSPATH elements. Automatically insert capacity-scheduler. @rem set HADOOP_CLASSPATH= I noticed that there is no equivalent "@rem set JAVA_HOME" line.
          Hide
          Ivan Mitic added a comment -

          Thanks Chris for reviewing the patch!

          1.hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time.

          Fixed.
          2.hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER?
          This is equivalent to what happens in hadoop sh script, right?
          3.hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
          Not sure I fully understand. Do you want to have a line set JAVA_HOME=%JAVA_HOME%?

          Show
          Ivan Mitic added a comment - Thanks Chris for reviewing the patch! 1.hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time. Fixed. 2.hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER? This is equivalent to what happens in hadoop sh script, right? 3.hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here? Not sure I fully understand. Do you want to have a line set JAVA_HOME=%JAVA_HOME% ?
          Hide
          Ivan Mitic added a comment -

          Addressing Chris' comments.

          Show
          Ivan Mitic added a comment - Addressing Chris' comments.
          Hide
          Ivan Mitic added a comment -

          Minor updates to the patch to better align with yarn scripts.

          Show
          Ivan Mitic added a comment - Minor updates to the patch to better align with yarn scripts.
          Hide
          Chris Nauroth added a comment -

          Hello, Ivan. A few comments:

          1. hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time.
          2. hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER?
          3. hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
          Show
          Chris Nauroth added a comment - Hello, Ivan. A few comments: hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time. hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER? hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
          Hide
          Ivan Mitic added a comment -

          Attaching updated patch. Including changes to dist xml files so that scripts are properly binplaced in the distro.

          Show
          Ivan Mitic added a comment - Attaching updated patch. Including changes to dist xml files so that scripts are properly binplaced in the distro.
          Hide
          Ivan Mitic added a comment -

          Minor update to the patch.

          Show
          Ivan Mitic added a comment - Minor update to the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551845/HADOOP-9005.branch-trunk-win.scripts.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551845/HADOOP-9005.branch-trunk-win.scripts.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//console This message is automatically generated.
          Hide
          Ivan Mitic added a comment -

          Minor improvements to the patch.

          Show
          Ivan Mitic added a comment - Minor improvements to the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551842/HADOOP-9005.branch-trunk-win.scripts.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551842/HADOOP-9005.branch-trunk-win.scripts.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//console This message is automatically generated.
          Hide
          Ivan Mitic added a comment -

          Attaching the patch.

          Contributed by David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.

          Show
          Ivan Mitic added a comment - Attaching the patch. Contributed by David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.

            People

            • Assignee:
              Ivan Mitic
              Reporter:
              Ivan Mitic
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development