Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9005

Merge hadoop cmd line scripts from branch-1-win

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • trunk-win
    • trunk-win
    • None
    • None
    • Reviewed

    Description

      Tracking Jira for merging hadoop cmd line scripts from branch-1-win to trunk. Scripts also have to be updated to reflect their unix equivalents.

      Attachments

        Issue Links

          Activity

            ivanmi Ivan Mitic added a comment -

            Attaching the patch.

            Contributed by David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.

            ivanmi Ivan Mitic added a comment - Attaching the patch. Contributed by David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12551842/HADOOP-9005.branch-trunk-win.scripts.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551842/HADOOP-9005.branch-trunk-win.scripts.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//console This message is automatically generated.
            ivanmi Ivan Mitic added a comment -

            Minor improvements to the patch.

            ivanmi Ivan Mitic added a comment - Minor improvements to the patch.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12551845/HADOOP-9005.branch-trunk-win.scripts.2.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551845/HADOOP-9005.branch-trunk-win.scripts.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//console This message is automatically generated.
            ivanmi Ivan Mitic added a comment -

            Minor update to the patch.

            ivanmi Ivan Mitic added a comment - Minor update to the patch.
            ivanmi Ivan Mitic added a comment -

            Attaching updated patch. Including changes to dist xml files so that scripts are properly binplaced in the distro.

            ivanmi Ivan Mitic added a comment - Attaching updated patch. Including changes to dist xml files so that scripts are properly binplaced in the distro.
            cnauroth Chris Nauroth added a comment -

            Hello, Ivan. A few comments:

            1. hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time.
            2. hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER?
            3. hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
            cnauroth Chris Nauroth added a comment - Hello, Ivan. A few comments: hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time. hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER? hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
            ivanmi Ivan Mitic added a comment -

            Minor updates to the patch to better align with yarn scripts.

            ivanmi Ivan Mitic added a comment - Minor updates to the patch to better align with yarn scripts.
            ivanmi Ivan Mitic added a comment -

            Addressing Chris' comments.

            ivanmi Ivan Mitic added a comment - Addressing Chris' comments.
            ivanmi Ivan Mitic added a comment -

            Thanks Chris for reviewing the patch!

            1.hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time.

            Fixed.
            2.hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER?
            This is equivalent to what happens in hadoop sh script, right?
            3.hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
            Not sure I fully understand. Do you want to have a line set JAVA_HOME=%JAVA_HOME%?

            ivanmi Ivan Mitic added a comment - Thanks Chris for reviewing the patch! 1.hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time. Fixed. 2.hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER? This is equivalent to what happens in hadoop sh script, right? 3.hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here? Not sure I fully understand. Do you want to have a line set JAVA_HOME=%JAVA_HOME% ?
            cnauroth Chris Nauroth added a comment -

            2. I see lines like this in hadoop-env.sh. It seems audit logger is disabled instead of security logger.

            export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
            export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
            

            3. Most of the variables look to be set up with commented out lines for people to uncomment and use. i.e.:

            @rem Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
            @rem set HADOOP_CLASSPATH=
            

            I noticed that there is no equivalent "@rem set JAVA_HOME" line.

            cnauroth Chris Nauroth added a comment - 2. I see lines like this in hadoop-env.sh. It seems audit logger is disabled instead of security logger. export HADOOP_NAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" 3. Most of the variables look to be set up with commented out lines for people to uncomment and use. i.e.: @rem Extra Java CLASSPATH elements. Automatically insert capacity-scheduler. @rem set HADOOP_CLASSPATH= I noticed that there is no equivalent "@rem set JAVA_HOME" line.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//console This message is automatically generated.
            ivanmi Ivan Mitic added a comment -

            Thanks for clarifying Chris!

            2. Hmm, I still don't see the problem.
            This is the snippet from the shell script:

            # Command specific options appended to HADOOP_OPTS when specified
            export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
            export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
            export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
            

            And this is the cmd script equivalent:

            @rem Command specific options appended to HADOOP_OPTS when specified
            if not defined HADOOP_SECURITY_LOGGER (
              set HADOOP_SECURITY_LOGGER=INFO,RFAS
            )
            if not defined HDFS_AUDIT_LOGGER (
              set HDFS_AUDIT_LOGGER=INFO,NullAppender
            )
            
            set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS%
            set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS%
            set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS%
            

            3. Sure, I'll fix this.

            ivanmi Ivan Mitic added a comment - Thanks for clarifying Chris! 2. Hmm, I still don't see the problem. This is the snippet from the shell script: # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_DATANODE_OPTS= "-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS= "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" And this is the cmd script equivalent: @rem Command specific options appended to HADOOP_OPTS when specified if not defined HADOOP_SECURITY_LOGGER ( set HADOOP_SECURITY_LOGGER=INFO,RFAS ) if not defined HDFS_AUDIT_LOGGER ( set HDFS_AUDIT_LOGGER=INFO,NullAppender ) set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS% set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS% set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS% 3. Sure, I'll fix this.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//console This message is automatically generated.
            cnauroth Chris Nauroth added a comment -

            2. Please disregard this comment. My original comment was in reference to hadoop.cmd (not hadoop-env.cmd), but I see now that this is equivalent to the sh version anyway. Thank you.

            cnauroth Chris Nauroth added a comment - 2. Please disregard this comment. My original comment was in reference to hadoop.cmd (not hadoop-env.cmd), but I see now that this is equivalent to the sh version anyway. Thank you.
            ivanmi Ivan Mitic added a comment -

            Attaching updated patch.

            The patch also includes basic start/stop scripts so that we can unblock getting trunk running on Windows. Start/stop scripts are not complete at this point, that will be addressed thru a separate Jira.

            ivanmi Ivan Mitic added a comment - Attaching updated patch. The patch also includes basic start/stop scripts so that we can unblock getting trunk running on Windows. Start/stop scripts are not complete at this point, that will be addressed thru a separate Jira.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12552361/HADOOP-9005.branch-trunk-win.scripts.7.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12552361/HADOOP-9005.branch-trunk-win.scripts.7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//console This message is automatically generated.
            cnauroth Chris Nauroth added a comment -

            +1 for the latest patch

            I tested successfully on my Windows VM. Thank you for taking care of the assembly definitions too.

            cnauroth Chris Nauroth added a comment - +1 for the latest patch I tested successfully on my Windows VM. Thank you for taking care of the assembly definitions too.

            +1

            Thank you Ivan. I committed the patch to branch-trunk-win.

            sureshms Suresh Srinivas added a comment - +1 Thank you Ivan. I committed the patch to branch-trunk-win.

            Missed other contributors to this patch. Thank you David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.

            sureshms Suresh Srinivas added a comment - Missed other contributors to this patch. Thank you David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.

            People

              ivanmi Ivan Mitic
              ivanmi Ivan Mitic
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: