Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3916

various issues with running yarn proxyserver

    Details

      Description

      Seem like yarn proxyserver is not operational when running out of the 0.23.1 RC2 tarball.

      1. Setting yarn.web-proxy.address to match yarn.resourcemanager.address doesn't disable the proxyserver (althought not setting yarn.web-proxy.address at all correctly disable it and produces a message: org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run). This contradicts the documentation provided for yarn.web-proxy.address in yarn-default.xml
      1. Setting yarn.web-proxy.address and running the service results in the following:
      $ ./sbin/yarn-daemon.sh start proxyserver 
      starting proxyserver, logging to /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
      /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
      

      with the following message found in the logs:

      2012-02-24 09:26:31,099 FATAL org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web server
      java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
              at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
              at org.apache.hadoop.http.HttpServer.<init>(HttpServer.java:224)
              at org.apache.hadoop.http.HttpServer.<init>(HttpServer.java:164)
              at org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
              at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
              at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
      

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          Maybe svn add was not done before creating the patch? Since Jenkins did not have the right patch, hopefully this did not cause any issues.

          Show
          Suresh Srinivas added a comment - Maybe svn add was not done before creating the patch? Since Jenkins did not have the right patch, hopefully this did not cause any issues.
          Hide
          Harsh J added a comment -

          Hi Suresh,

          The patch attached here may have ignored the dotfile. The change can be properly seen at http://svn.apache.org/viewvc?view=revision&revision=1307413 so its not a doc-only fix.

          Show
          Harsh J added a comment - Hi Suresh, The patch attached here may have ignored the dotfile. The change can be properly seen at http://svn.apache.org/viewvc?view=revision&revision=1307413 so its not a doc-only fix.
          Hide
          Suresh Srinivas added a comment -

          Alejandro, all I see in this patch is just a change in the description in yarn-default.xml. How does this solve the problem? If it is just a doc update, should the title of this jira be updated?

          Show
          Suresh Srinivas added a comment - Alejandro, all I see in this patch is just a change in the description in yarn-default.xml. How does this solve the problem? If it is just a doc update, should the title of this jira be updated?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1036 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1036/)
          MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1036 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1036/ ) MAPREDUCE-3916 . various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1001 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1001/)
          MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1001 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1001/ ) MAPREDUCE-3916 . various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1967 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1967/)
          MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413)

          Result = ABORTED
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1967 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1967/ ) MAPREDUCE-3916 . various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1954 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1954/)
          MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1954 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1954/ ) MAPREDUCE-3916 . various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2029/)
          MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2029 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2029/ ) MAPREDUCE-3916 . various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          Devaraj K added a comment -

          Thanks a lot Alejandro for reviewing and verifying the patch.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          No new tests required because changes done in yarn-default.xml and added one dir.

          Show
          Devaraj K added a comment - Thanks a lot Alejandro for reviewing and verifying the patch. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. No new tests required because changes done in yarn-default.xml and added one dir.
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Devaraj. I had to manually create the proxy/.keep file as the patch command ignored it. Committed to trunk and branch-2

          Show
          Alejandro Abdelnur added a comment - Thanks Devaraj. I had to manually create the proxy/.keep file as the patch command ignored it. Committed to trunk and branch-2
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520600/MAPREDUCE-3916.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520600/MAPREDUCE-3916.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1. built/deployed pseudo cluster, set yarn.web-proxy.address and proxy started successfully

          Show
          Alejandro Abdelnur added a comment - +1. built/deployed pseudo cluster, set yarn.web-proxy.address and proxy started successfully
          Hide
          Devaraj K added a comment -

          Setting yarn.web-proxy.address to match yarn.resourcemanager.address doesn't disable the proxyserver (althought not setting yarn.web-proxy.address at all correctly disable it and produces a message: org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run). This contradicts the documentation provided for yarn.web-proxy.address in yarn-default.xml

          If we don't set 'yarn.web-proxy.address', then the proxy server will not start as separate process and start up fails with the message 'org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run' and proxy will run as part of the RM.

          If we give the value for 'yarn.web-proxy.address' which matches to the 'yarn.resourcemanager.address' then proxy server start up fails with "Address already in use". And also it is not starting as part of the RM. It is not as per the documentation. I have modified the description for this property accordingly.

          Setting yarn.web-proxy.address and running the service results in the following:

          I have added the 'hadoop-yarn-common\src\main\resources\webapps\proxy' dir with .keep file in it to resolve this problem as done for other components.

          Show
          Devaraj K added a comment - Setting yarn.web-proxy.address to match yarn.resourcemanager.address doesn't disable the proxyserver (althought not setting yarn.web-proxy.address at all correctly disable it and produces a message: org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run). This contradicts the documentation provided for yarn.web-proxy.address in yarn-default.xml If we don't set 'yarn.web-proxy.address', then the proxy server will not start as separate process and start up fails with the message 'org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run' and proxy will run as part of the RM. If we give the value for 'yarn.web-proxy.address' which matches to the 'yarn.resourcemanager.address' then proxy server start up fails with "Address already in use". And also it is not starting as part of the RM. It is not as per the documentation. I have modified the description for this property accordingly. Setting yarn.web-proxy.address and running the service results in the following: I have added the 'hadoop-yarn-common\src\main\resources\webapps\proxy' dir with .keep file in it to resolve this problem as done for other components.

            People

            • Assignee:
              Devaraj K
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development