Details
Description
Tracking Jira for merging hadoop cmd line scripts from branch-1-win to trunk. Scripts also have to be updated to reflect their unix equivalents.
Attachments
Attachments
- HADOOP-9005.branch-trunk-win.scripts.7.patch
- 25 kB
- Ivan Mitic
- HADOOP-9005.branch-trunk-win.scripts.6.patch
- 20 kB
- Ivan Mitic
- HADOOP-9005.branch-trunk-win.scripts.5.patch
- 20 kB
- Ivan Mitic
- HADOOP-9005.branch-trunk-win.scripts.4.patch
- 20 kB
- Ivan Mitic
- HADOOP-9005.branch-trunk-win.scripts.3.patch
- 18 kB
- Ivan Mitic
- HADOOP-9005.branch-trunk-win.scripts.2.patch
- 18 kB
- Ivan Mitic
- HADOOP-9005.branch-trunk-win.scripts.patch
- 18 kB
- Ivan Mitic
Issue Links
- is part of
-
HADOOP-8562 Enhancements to support Hadoop on Windows Server and Windows Azure environments
- Closed
Activity
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551842/HADOOP-9005.branch-trunk-win.scripts.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1697//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551845/HADOOP-9005.branch-trunk-win.scripts.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1698//console
This message is automatically generated.
Attaching updated patch. Including changes to dist xml files so that scripts are properly binplaced in the distro.
Hello, Ivan. A few comments:
- hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time.
- hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER?
- hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
Thanks Chris for reviewing the patch!
1.hadoop-config.cmd line 156: This might be easier with an else clause instead of checking defined HADOOP_USER_CLASSPATH_FIRST a second time.
Fixed.
2.hadoop.cmd line 138: Why is it important to disable HADOOP_SECURITY_LOGGER?
This is equivalent to what happens in hadoop sh script, right?
3.hadoop-env.cmd line 25: Should we add a commented out set JAVA_HOME line here?
Not sure I fully understand. Do you want to have a line set JAVA_HOME=%JAVA_HOME%?
2. I see lines like this in hadoop-env.sh. It seems audit logger is disabled instead of security logger.
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
3. Most of the variables look to be set up with commented out lines for people to uncomment and use. i.e.:
@rem Extra Java CLASSPATH elements. Automatically insert capacity-scheduler. @rem set HADOOP_CLASSPATH=
I noticed that there is no equivalent "@rem set JAVA_HOME" line.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1700//console
This message is automatically generated.
Thanks for clarifying Chris!
2. Hmm, I still don't see the problem.
This is the snippet from the shell script:
# Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
And this is the cmd script equivalent:
@rem Command specific options appended to HADOOP_OPTS when specified if not defined HADOOP_SECURITY_LOGGER ( set HADOOP_SECURITY_LOGGER=INFO,RFAS ) if not defined HDFS_AUDIT_LOGGER ( set HDFS_AUDIT_LOGGER=INFO,NullAppender ) set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS% set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS% set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS%
3. Sure, I'll fix this.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551939/HADOOP-9005.branch-trunk-win.scripts.6.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1702//console
This message is automatically generated.
2. Please disregard this comment. My original comment was in reference to hadoop.cmd (not hadoop-env.cmd), but I see now that this is equivalent to the sh version anyway. Thank you.
Attaching updated patch.
The patch also includes basic start/stop scripts so that we can unblock getting trunk running on Windows. Start/stop scripts are not complete at this point, that will be addressed thru a separate Jira.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12552361/HADOOP-9005.branch-trunk-win.scripts.7.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-assemblies hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1716//console
This message is automatically generated.
+1 for the latest patch
I tested successfully on my Windows VM. Thank you for taking care of the assembly definitions too.
Missed other contributors to this patch. Thank you David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.
Attaching the patch.
Contributed by David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic.