Hadoop Common
  1. Hadoop Common
  2. HADOOP-10050

Update single node and cluster install instructions to work with latest bits

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: documentation
    • Labels:
      None
    • Target Version/s:

      Description

      A few things i noticed

      1. changes to yarn.nodemanager.aux-services
      2. Set the framework to yarn in mapred-site.xml
      3. Start the history server

      Also noticed no change to the capacity scheduler configs was needed.

      1. ClusterSetup.html
        66 kB
        Arpit Gupta
      2. HADOOP-10050.patch
        6 kB
        Arpit Gupta
      3. HADOOP-10050.patch
        5 kB
        Arpit Gupta
      4. mapred-site.xml
        1 kB
        Arpit Gupta
      5. SingleCluster.html
        19 kB
        Arpit Gupta
      6. yarn-site.xml
        2 kB
        Arpit Gupta

        Issue Links

          Activity

          Hide
          Sandy Ryza added a comment -

          We should include the changes proposed in YARN-1227 with this.

          Show
          Sandy Ryza added a comment - We should include the changes proposed in YARN-1227 with this.
          Hide
          Arpit Gupta added a comment -

          Sandy Ryza

          Yup that was already part of the config clean up i was going to do . Should we resolve it as a dup?

          Show
          Arpit Gupta added a comment - Sandy Ryza Yup that was already part of the config clean up i was going to do . Should we resolve it as a dup?
          Hide
          Arpit Gupta added a comment -

          Attached are the configs i ended up using. If there are any other changes we would like to make let me know and i can address them as well.

          Show
          Arpit Gupta added a comment - Attached are the configs i ended up using. If there are any other changes we would like to make let me know and i can address them as well.
          Hide
          Arpit Gupta added a comment -

          Attached an initial patch.

          Also i would like to

          hadoop-mapreduce-project/conf/mapred-site.xml.template -> hadoop-mapreduce-project/conf/mapred-site.xml

          Not sure how a rename patch is generated. This is needed so that the file generated does not have to be renamed .

          Show
          Arpit Gupta added a comment - Attached an initial patch. Also i would like to hadoop-mapreduce-project/conf/mapred-site.xml.template -> hadoop-mapreduce-project/conf/mapred-site.xml Not sure how a rename patch is generated. This is needed so that the file generated does not have to be renamed .
          Hide
          Arpit Gupta added a comment -

          Attaching the html files for the docs in case some one wants to take a look.

          Show
          Arpit Gupta added a comment - Attaching the html files for the docs in case some one wants to take a look.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12608652/HADOOP-10050.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3226//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3226//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608652/HADOOP-10050.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3226//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3226//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12608652/HADOOP-10050.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3227//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3227//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608652/HADOOP-10050.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3227//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3227//console This message is automatically generated.
          Hide
          Hitesh Shah added a comment -

          Comments:

          The capacity-scheduler configs are removed from SingleCluster.apt.vm - any reason for this?
          Generated SingleCluster.html has "Set $$YARN_CONF_DIR" - mind fixing this too?

          Run ResourceManager and NodeManager as:
          
          $ cd $HADOOP_MAPRED_HOME
          $ sbin/yarn-daemon.sh start resourcemanager
          $ sbin/yarn-daemon.sh start nodemanager
          $ sbin/mr-jobhistory-daemon.sh start historyserver
          

          yarn-daemon.sh is not meant to be in HADOOP_MAPRED_HOME.

          If this doc needs to be fixed, there are probably a lot of confusing references to HADOOP_PREFIX vs HADOOP_YARN_HOME etc which probably need to be ironed out. It may be good to either address them in this jira or file a new one for a follow-up.

          Show
          Hitesh Shah added a comment - Comments: The capacity-scheduler configs are removed from SingleCluster.apt.vm - any reason for this? Generated SingleCluster.html has "Set $$YARN_CONF_DIR" - mind fixing this too? Run ResourceManager and NodeManager as: $ cd $HADOOP_MAPRED_HOME $ sbin/yarn-daemon.sh start resourcemanager $ sbin/yarn-daemon.sh start nodemanager $ sbin/mr-jobhistory-daemon.sh start historyserver yarn-daemon.sh is not meant to be in HADOOP_MAPRED_HOME. If this doc needs to be fixed, there are probably a lot of confusing references to HADOOP_PREFIX vs HADOOP_YARN_HOME etc which probably need to be ironed out. It may be good to either address them in this jira or file a new one for a follow-up.
          Hide
          Arpit Gupta added a comment -

          The capacity-scheduler configs are removed from SingleCluster.apt.vm - any reason for this?

          I did not have to make any changes to be able to run an example.

          We have yarn.scheduler.capacity.root.default.capacity=100 and yarn.scheduler.capacity.root.queues=default.

          The following 2 are not there in the configs that are part of the release yarn.scheduler.capacity.root.capacity and yarn.scheduler.capacity.root.unfunded.capacity

          Generated SingleCluster.html has "Set $$YARN_CONF_DIR" - mind fixing this too?

          Will fix this.

          yarn-daemon.sh is not meant to be in HADOOP_MAPRED_HOME.

          Actually when i ran the hadoop-2.2.0 tar ball i did not have to set any of these env vars everything was in ./sbin. I can make the change to to HADOOP_YARN_HOME and HADOOP_MAPRED_HOME.

          I cannot find any reference for HADOOP_PREFIX on http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html

          Show
          Arpit Gupta added a comment - The capacity-scheduler configs are removed from SingleCluster.apt.vm - any reason for this? I did not have to make any changes to be able to run an example. We have yarn.scheduler.capacity.root.default.capacity=100 and yarn.scheduler.capacity.root.queues=default. The following 2 are not there in the configs that are part of the release yarn.scheduler.capacity.root.capacity and yarn.scheduler.capacity.root.unfunded.capacity Generated SingleCluster.html has "Set $$YARN_CONF_DIR" - mind fixing this too? Will fix this. yarn-daemon.sh is not meant to be in HADOOP_MAPRED_HOME. Actually when i ran the hadoop-2.2.0 tar ball i did not have to set any of these env vars everything was in ./sbin. I can make the change to to HADOOP_YARN_HOME and HADOOP_MAPRED_HOME. I cannot find any reference for HADOOP_PREFIX on http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html
          Hide
          Arpit Gupta added a comment -

          Addressed everything except for capacity scheduler changes. Let me know if we want to put those back.

          Also found a minor issue on the last line of the page

          http://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html

          where the property should have been mapred_shufflex instead of mapred.shufflex so fixed that as well.

          Show
          Arpit Gupta added a comment - Addressed everything except for capacity scheduler changes. Let me know if we want to put those back. Also found a minor issue on the last line of the page http://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html where the property should have been mapred_shufflex instead of mapred.shufflex so fixed that as well.
          Hide
          Sandy Ryza added a comment -

          I'm not sure that the yarn.resourcemanager.hostname port should be set to 0.0.0.0. 0.0.0.0 only makes sense as a port to bind to, and this config is also used by clients to connect to the resourcemanager.

          Show
          Sandy Ryza added a comment - I'm not sure that the yarn.resourcemanager.hostname port should be set to 0.0.0.0. 0.0.0.0 only makes sense as a port to bind to, and this config is also used by clients to connect to the resourcemanager.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12608962/HADOOP-10050.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3233//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3233//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608962/HADOOP-10050.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3233//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3233//console This message is automatically generated.
          Hide
          Arpit Gupta added a comment -

          Actually for my single node setup i used 0.0.0.0 for the property and it worked better. With it set to 0.0.0.0 the Application Master links for the running job worked where as when i set it to localhost those links did not work. If i changed the ip to localhost it would bring up the page.

          So i think leaving to 0.0.0.0 might be better unless i am missing some configs.

          Show
          Arpit Gupta added a comment - Actually for my single node setup i used 0.0.0.0 for the property and it worked better. With it set to 0.0.0.0 the Application Master links for the running job worked where as when i set it to localhost those links did not work. If i changed the ip to localhost it would bring up the page. So i think leaving to 0.0.0.0 might be better unless i am missing some configs.
          Hide
          Arpit Gupta added a comment -

          Anybody have any thoughts on what config i am missing that causes the running AM link to not work?

          Show
          Arpit Gupta added a comment - Anybody have any thoughts on what config i am missing that causes the running AM link to not work?
          Hide
          Allen Wittenauer added a comment -

          There are a lot of good changes in this patch, but with HADOOP-9902 committed, it's out of date. See HADOOP-10908 for the quick list I made.

          Show
          Allen Wittenauer added a comment - There are a lot of good changes in this patch, but with HADOOP-9902 committed, it's out of date. See HADOOP-10908 for the quick list I made.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12608962/HADOOP-10050.patch
          against trunk revision 6b7673e.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4815//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12608962/HADOOP-10050.patch against trunk revision 6b7673e. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4815//console This message is automatically generated.
          Hide
          Allen Wittenauer added a comment - - edited

          Cancelling patch. These files have changed a lot since this patch was created.

          Show
          Allen Wittenauer added a comment - - edited Cancelling patch. These files have changed a lot since this patch was created.

            People

            • Assignee:
              Arpit Gupta
              Reporter:
              Arpit Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development