Uploading a starter patch

This patch moves historyserver start/stop to bin/mapred from bin/yarn. I have added support scripts bin/start-mapred.sh bin/stop-mapred.sh and bin/mapred-daemon.sh as supporting scripts. All three support scripts are based completely on their yarn counter parts. I have opted at this point not to create corresponding MAPRED_ environment under the assumption that mapreduce always depends on yarn. I have verified that controlling history server from start-mapred.sh, stop-mapred.sh, mapred-daemon.sh, and mapred all work as expected.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507248/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1437//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1437//console

This message is automatically generated.

Sorry Jon,
The patch no longer applies after MAPREDUCE-3366 commit. Can you please regenerate the patch?

Upmerged the patch

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507711/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1470//console

This message is automatically generated.

With the assemblies change, patch will fail to apply. Only change from the upmerge was to change files to apply patch to (start/stop-all -> start/stop-yarn)

Jon,
While you are at it, can you please clean up bin/mapred as well? It has help statements for jobtracker/tasktracker and still commands for acting upon "jobtracker"/"tasktracker" commands.

Absolutely. Every little bit helps.

Addressing @Mahadev's comments

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509318/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1525//console

This message is automatically generated.

Patch modifies hadoop-assemblies so patch apply failure is expected.

Jon, if you don't mind, can you please update the docs as well? Thanks.

Thanks, Arun. Will take care of this.

Updated patch to address ClusterSetup.apt.vm and hadoop-mapreduce-project/INSTALL aligning the documentation with the new method of starting the history server.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509693/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1556//console

This message is automatically generated.

Patch modifies hadoop-assemblies so patch apply failure is expected.

Jonathan,
minor comments from the patch.

usage="Usage: mapreduce-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) <yarn-command> "

I think, instead of mapreduce-daemon.sh, it should be mapred-daemon.sh in mapred-daemon.sh. Moreover after inclusion of this patch, it will be nomore a yarn-command, it will be a mapred-command.

$ $YARN_HOME/bin/mapred stop historyserver --config $HADOOP_CONF_DIR 

I think, we should replace HADOOP_CONF_DIR by YARN_CONF_DIR and mapred by mapred-daemon.sh in ClusterSetup.apt.vm.

Thanks for the comments. I have fixed the issues you have addressed. I have left historyserver being started by mapred since i think that better reflects the recommended usage.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509929/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1570//console

This message is automatically generated.

Patch modifies hadoop-assemblies so patch apply failure is expected.

Upmerging to the patch

Verified that mapred-daemon.sh can both start and stop the job history server. Verified that start-mapred.sh and stop-mapred.sh also work.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513502/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1798//console

This message is automatically generated.

Patch failure is expected due to hadoop-assemblies change.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513514/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1801//console

This message is automatically generated.

I just committed this. Thanks Jonathan!

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513517/MAPREDUCE-3354.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1802//console

This message is automatically generated.

Thanks, Arun.

Integrated in Hadoop-Hdfs-0.23-Commit #490 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/490/)
Merge -c 1241250 from trunk to branch-0.23 to fix MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241252
Files :

• /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/INSTALL
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Common-trunk-Commit #1674 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1674/)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241250
Files :

• /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Common-0.23-Commit #501 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/501/)
Merge -c 1241250 from trunk to branch-0.23 to fix MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241252
Files :

• /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/INSTALL
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Hdfs-trunk-Commit #1747 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1747/)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241250
Files :

• /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Mapreduce-0.23-Commit #510 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/510/)
Merge -c 1241250 from trunk to branch-0.23 to fix MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241252
Files :

• /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/INSTALL
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Mapreduce-trunk-Commit #1688 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1688/)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241250
Files :

• /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241250
Files :

• /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/)
Merge -c 1241250 from trunk to branch-0.23 to fix MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241252
Files :

• /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/INSTALL
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/)
Merge -c 1241250 from trunk to branch-0.23 to fix MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241252
Files :

• /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/INSTALL
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm

Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/)
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by bin/mapred instead of bin/yarn. Contributed by Jonathan Eagles.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241250
Files :

• /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred
• /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/start-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/stop-yarn.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm