Hadoop YARN
  1. Hadoop YARN
  2. YARN-800

Clicking on an AM link for a running app leads to a HTTP 500

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Clicking the AM link tries to open up a page with url like

      http://hostname:8088/proxy/application_1370886527995_0645/

      and this leads to an HTTP 500

        Issue Links

          Activity

          Arpit Gupta created issue -
          Hide
          Arpit Gupta added a comment -

          Here is the stack trace

          HTTP ERROR 500
          
          Problem accessing /proxy/application_1370886527995_0658/. Reason:
          
              Connection refused
          
          Caused by:
          
          java.net.ConnectException: Connection refused
          	at java.net.PlainSocketImpl.socketConnect(Native Method)
          	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
          	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
          	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
          	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
          	at java.net.Socket.connect(Socket.java:529)
          	at java.net.Socket.connect(Socket.java:478)
          	at java.net.Socket.<init>(Socket.java:375)
          	at java.net.Socket.<init>(Socket.java:249)
          	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
          	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
          	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
          	at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185)
          	at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:334)
          	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
          	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
          	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
          	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
          	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
          	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
          	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
          	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
          	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
          	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
          	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
          	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
          	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
          	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
          	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
          	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1077)
          	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
          	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
          	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
          	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
          	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
          	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
          	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
          	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
          	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
          	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
          	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
          	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          	at org.mortbay.jetty.Server.handle(Server.java:326)
          	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
          	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
          	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
          	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
          	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
          	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
          	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
          
          Powered by Jetty://
          
          Show
          Arpit Gupta added a comment - Here is the stack trace HTTP ERROR 500 Problem accessing /proxy/application_1370886527995_0658/. Reason: Connection refused Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at java.net.Socket.<init>(Socket.java:375) at java.net.Socket.<init>(Socket.java:249) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1077) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Powered by Jetty: //
          Hide
          Arpit Gupta added a comment -

          Looks like we have to set the property yarn.resourcemanager.webapp.address to RMAddress:8088 which should not be the case. We should be defaulting the appropriate value in the system.

          Show
          Arpit Gupta added a comment - Looks like we have to set the property yarn.resourcemanager.webapp.address to RMAddress:8088 which should not be the case. We should be defaulting the appropriate value in the system.
          Hide
          Zhijie Shen added a comment -

          Did a quick local test, and found the link was not broken. It seems that the default value has already been in yarn-default.xml

            <property>
              <description>The hostname of the RM.</description>
              <name>yarn.resourcemanager.hostname</name>
              <value>0.0.0.0</value>
            </property> 
          
            <property>
              <description>The address of the RM web application.</description>
              <name>yarn.resourcemanager.webapp.address</name>
              <value>${yarn.resourcemanager.hostname}:8088</value>
            </property>
          

          and YarnConfiguration

            public static final String RM_WEBAPP_ADDRESS = 
              RM_PREFIX + "webapp.address";
          
            public static final int DEFAULT_RM_WEBAPP_PORT = 8088;
            public static final String DEFAULT_RM_WEBAPP_ADDRESS = "0.0.0.0:" +
              DEFAULT_RM_WEBAPP_PORT;
          

          Looked into the code, it seems to be related to "yarn.web-proxy.address". In WebAppProxyServlet,

                resp.setStatus(client.executeMethod(config, method));
          

          tries to connect the proxy host to show the application webpage. If "yarn.web-proxy.address" is not set, RM will become the proxy, and its address will be ${yarn.resourcemanager.hostname}:8088 as well.

          Maybe it is good to check the configuration of "yarn.web-proxy.address"

          Show
          Zhijie Shen added a comment - Did a quick local test, and found the link was not broken. It seems that the default value has already been in yarn-default.xml <property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>0.0.0.0</value> </property> <property> <description>The address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property> and YarnConfiguration public static final String RM_WEBAPP_ADDRESS = RM_PREFIX + "webapp.address" ; public static final int DEFAULT_RM_WEBAPP_PORT = 8088; public static final String DEFAULT_RM_WEBAPP_ADDRESS = "0.0.0.0:" + DEFAULT_RM_WEBAPP_PORT; Looked into the code, it seems to be related to "yarn.web-proxy.address". In WebAppProxyServlet, resp.setStatus(client.executeMethod(config, method)); tries to connect the proxy host to show the application webpage. If "yarn.web-proxy.address" is not set, RM will become the proxy, and its address will be ${yarn.resourcemanager.hostname}:8088 as well. Maybe it is good to check the configuration of "yarn.web-proxy.address"
          Hide
          Jian He added a comment -

          The link is not broken in local cluster because by default local cluster use localhost(0.0.0.0) as RM Ip Address.
          This problem occurs on real cluster where RM is using real IP address, and RM_WEBAPP_ADDRESS is still defaulting to '0.0.0.0', we should default RM_WEBAPP_ADDRESS to RM_ADDRESS.
          If we specify RM_WEBAPP_ADDRESS explicitly, this problem will not occur

          Show
          Jian He added a comment - The link is not broken in local cluster because by default local cluster use localhost(0.0.0.0) as RM Ip Address. This problem occurs on real cluster where RM is using real IP address, and RM_WEBAPP_ADDRESS is still defaulting to '0.0.0.0', we should default RM_WEBAPP_ADDRESS to RM_ADDRESS. If we specify RM_WEBAPP_ADDRESS explicitly, this problem will not occur
          Arpit Gupta made changes -
          Field Original Value New Value
          Priority Critical [ 2 ] Minor [ 4 ]
          Hide
          Arpit Gupta added a comment -

          Changing it to minor. If i set yarn.resourcemanager.hostname webapp adress default gets picked up correctly. Leaving it open to see if we can have better default resolution once YARN-673 gets resolved. I think we could check the hostname property as well as the rm address property and pick one of those if only one of them is set.

          Show
          Arpit Gupta added a comment - Changing it to minor. If i set yarn.resourcemanager.hostname webapp adress default gets picked up correctly. Leaving it open to see if we can have better default resolution once YARN-673 gets resolved. I think we could check the hostname property as well as the rm address property and pick one of those if only one of them is set.
          Hide
          Dave Disser added a comment -

          I'm seeing this issue regardless of the status of yarn.resourcemanager.hostname and yarn.web-proxy.address. I notice the following in my nodemanager log file:

          2014-05-28 14:06:20,478 INFO webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(330)) - dr.who is accessing
          unchecked http://hdp003-3:59959/ which is the app master GUI of application_1401300304842_0001 owned by hdfs

          I can try to retrieve this URL directly:

          hdp003-2:~ # wget -O - http://hdp003-3:59959/
          -2014-05-28 14:06:47- http://hdp003-3:59959/
          Resolving hdp003-3... 39.64.24.3
          Connecting to hdp003-3|39.64.24.3|:59959... connected.
          HTTP request sent, awaiting response... 302 Found
          Location: http://hdp003-3:59959/mapreduce [following]
          -2014-05-28 14:06:47- http://hdp003-3:59959/mapreduce
          Reusing existing connection to hdp003-3:59959.
          HTTP request sent, awaiting response... 302 Found
          Location: http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce [following]
          -2014-05-28 14:06:47- http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce
          Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused.
          Resolving hdp003-3... 39.64.24.3
          Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused.

          The node running the AM is proxying the request to itself, where there is no proxy running. If I do the same on the node where AM is running, I get the proper result:

          hdp003-3:~ # wget -O - http://hdp003-3:59959/
          -2014-05-28 14:07:25- http://hdp003-3:59959/
          Resolving hdp003-3... 39.64.24.3
          Connecting to hdp003-3|39.64.24.3|:59959... connected.
          HTTP request sent, awaiting response... 302 Found
          Location: http://hdp003-3:59959/mapreduce [following]
          -2014-05-28 14:07:25- http://hdp003-3:59959/mapreduce
          Reusing existing connection to hdp003-3:59959.
          HTTP request sent, awaiting response... 200 OK
          Length: 6224 (6.1K) [text/html]
          Saving to: `STDOUT'

          0% [ ] 0 --.-K/s <
          !DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
          <html>
          <meta http-equiv="Content-type" content="text/html; charset=UTF-8">
          <style type="text/css">
          ...

          Show
          Dave Disser added a comment - I'm seeing this issue regardless of the status of yarn.resourcemanager.hostname and yarn.web-proxy.address. I notice the following in my nodemanager log file: 2014-05-28 14:06:20,478 INFO webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(330)) - dr.who is accessing unchecked http://hdp003-3:59959/ which is the app master GUI of application_1401300304842_0001 owned by hdfs I can try to retrieve this URL directly: hdp003-2:~ # wget -O - http://hdp003-3:59959/ - 2014-05-28 14:06:47 - http://hdp003-3:59959/ Resolving hdp003-3... 39.64.24.3 Connecting to hdp003-3|39.64.24.3|:59959... connected. HTTP request sent, awaiting response... 302 Found Location: http://hdp003-3:59959/mapreduce [following] - 2014-05-28 14:06:47 - http://hdp003-3:59959/mapreduce Reusing existing connection to hdp003-3:59959. HTTP request sent, awaiting response... 302 Found Location: http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce [following] - 2014-05-28 14:06:47 - http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused. Resolving hdp003-3... 39.64.24.3 Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused. The node running the AM is proxying the request to itself, where there is no proxy running. If I do the same on the node where AM is running, I get the proper result: hdp003-3:~ # wget -O - http://hdp003-3:59959/ - 2014-05-28 14:07:25 - http://hdp003-3:59959/ Resolving hdp003-3... 39.64.24.3 Connecting to hdp003-3|39.64.24.3|:59959... connected. HTTP request sent, awaiting response... 302 Found Location: http://hdp003-3:59959/mapreduce [following] - 2014-05-28 14:07:25 - http://hdp003-3:59959/mapreduce Reusing existing connection to hdp003-3:59959. HTTP request sent, awaiting response... 200 OK Length: 6224 (6.1K) [text/html] Saving to: `STDOUT' 0% [ ] 0 --.-K/s < !DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <meta http-equiv="Content-type" content="text/html; charset=UTF-8"> <style type="text/css"> ...
          Hide
          Dave Disser added a comment -

          As a follow-up, I also notice that the proxying works correctly while the tracking URL is "UNASSIGNED" (the first couple seconds after AM container launch), but then HTTP 500 occurs shortly after.

          Show
          Dave Disser added a comment - As a follow-up, I also notice that the proxying works correctly while the tracking URL is "UNASSIGNED" (the first couple seconds after AM container launch), but then HTTP 500 occurs shortly after.
          Hide
          Matt Foley added a comment -

          Craig Welch fixed this as part of YARN-1994.

          Show
          Matt Foley added a comment - Craig Welch fixed this as part of YARN-1994 .
          Hide
          Matt Foley added a comment -

          Arpit Gupta, can you please mark it resolved? I lack privs to do so in this project.

          Show
          Matt Foley added a comment - Arpit Gupta , can you please mark it resolved? I lack privs to do so in this project.
          Hide
          Junping Du added a comment -

          Agree with Matt Foley's comments above, let's resolve this JIRA as duplicated with YARN-1994.

          Show
          Junping Du added a comment - Agree with Matt Foley 's comments above, let's resolve this JIRA as duplicated with YARN-1994 .
          Junping Du made changes -
          Link This issue duplicates YARN-1994 [ YARN-1994 ]
          Junping Du made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          660d 4h 3m 1 Junping Du 04/Apr/15 03:58

            People

            • Assignee:
              Unassigned
              Reporter:
              Arpit Gupta
            • Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development