Hadoop YARN
  1. Hadoop YARN
  2. YARN-800

Clicking on an AM link for a running app leads to a HTTP 500

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Clicking the AM link tries to open up a page with url like

      http://hostname:8088/proxy/application_1370886527995_0645/

      and this leads to an HTTP 500

        Activity

        Hide
        Arpit Gupta added a comment -

        Here is the stack trace

        HTTP ERROR 500
        
        Problem accessing /proxy/application_1370886527995_0658/. Reason:
        
            Connection refused
        
        Caused by:
        
        java.net.ConnectException: Connection refused
        	at java.net.PlainSocketImpl.socketConnect(Native Method)
        	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        	at java.net.Socket.connect(Socket.java:529)
        	at java.net.Socket.connect(Socket.java:478)
        	at java.net.Socket.<init>(Socket.java:375)
        	at java.net.Socket.<init>(Socket.java:249)
        	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
        	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
        	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
        	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
        	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
        	at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185)
        	at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:334)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
        	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
        	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
        	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
        	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1077)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        	at org.mortbay.jetty.Server.handle(Server.java:326)
        	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
        
        Powered by Jetty://
        
        Show
        Arpit Gupta added a comment - Here is the stack trace HTTP ERROR 500 Problem accessing /proxy/application_1370886527995_0658/. Reason: Connection refused Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at java.net.Socket.<init>(Socket.java:375) at java.net.Socket.<init>(Socket.java:249) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1077) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Powered by Jetty: //
        Hide
        Arpit Gupta added a comment -

        Looks like we have to set the property yarn.resourcemanager.webapp.address to RMAddress:8088 which should not be the case. We should be defaulting the appropriate value in the system.

        Show
        Arpit Gupta added a comment - Looks like we have to set the property yarn.resourcemanager.webapp.address to RMAddress:8088 which should not be the case. We should be defaulting the appropriate value in the system.
        Hide
        Zhijie Shen added a comment -

        Did a quick local test, and found the link was not broken. It seems that the default value has already been in yarn-default.xml

          <property>
            <description>The hostname of the RM.</description>
            <name>yarn.resourcemanager.hostname</name>
            <value>0.0.0.0</value>
          </property> 
        
          <property>
            <description>The address of the RM web application.</description>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>${yarn.resourcemanager.hostname}:8088</value>
          </property>
        

        and YarnConfiguration

          public static final String RM_WEBAPP_ADDRESS = 
            RM_PREFIX + "webapp.address";
        
          public static final int DEFAULT_RM_WEBAPP_PORT = 8088;
          public static final String DEFAULT_RM_WEBAPP_ADDRESS = "0.0.0.0:" +
            DEFAULT_RM_WEBAPP_PORT;
        

        Looked into the code, it seems to be related to "yarn.web-proxy.address". In WebAppProxyServlet,

              resp.setStatus(client.executeMethod(config, method));
        

        tries to connect the proxy host to show the application webpage. If "yarn.web-proxy.address" is not set, RM will become the proxy, and its address will be ${yarn.resourcemanager.hostname}:8088 as well.

        Maybe it is good to check the configuration of "yarn.web-proxy.address"

        Show
        Zhijie Shen added a comment - Did a quick local test, and found the link was not broken. It seems that the default value has already been in yarn-default.xml <property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>0.0.0.0</value> </property> <property> <description>The address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property> and YarnConfiguration public static final String RM_WEBAPP_ADDRESS = RM_PREFIX + "webapp.address" ; public static final int DEFAULT_RM_WEBAPP_PORT = 8088; public static final String DEFAULT_RM_WEBAPP_ADDRESS = "0.0.0.0:" + DEFAULT_RM_WEBAPP_PORT; Looked into the code, it seems to be related to "yarn.web-proxy.address". In WebAppProxyServlet, resp.setStatus(client.executeMethod(config, method)); tries to connect the proxy host to show the application webpage. If "yarn.web-proxy.address" is not set, RM will become the proxy, and its address will be ${yarn.resourcemanager.hostname}:8088 as well. Maybe it is good to check the configuration of "yarn.web-proxy.address"
        Hide
        Jian He added a comment -

        The link is not broken in local cluster because by default local cluster use localhost(0.0.0.0) as RM Ip Address.
        This problem occurs on real cluster where RM is using real IP address, and RM_WEBAPP_ADDRESS is still defaulting to '0.0.0.0', we should default RM_WEBAPP_ADDRESS to RM_ADDRESS.
        If we specify RM_WEBAPP_ADDRESS explicitly, this problem will not occur

        Show
        Jian He added a comment - The link is not broken in local cluster because by default local cluster use localhost(0.0.0.0) as RM Ip Address. This problem occurs on real cluster where RM is using real IP address, and RM_WEBAPP_ADDRESS is still defaulting to '0.0.0.0', we should default RM_WEBAPP_ADDRESS to RM_ADDRESS. If we specify RM_WEBAPP_ADDRESS explicitly, this problem will not occur
        Hide
        Arpit Gupta added a comment -

        Changing it to minor. If i set yarn.resourcemanager.hostname webapp adress default gets picked up correctly. Leaving it open to see if we can have better default resolution once YARN-673 gets resolved. I think we could check the hostname property as well as the rm address property and pick one of those if only one of them is set.

        Show
        Arpit Gupta added a comment - Changing it to minor. If i set yarn.resourcemanager.hostname webapp adress default gets picked up correctly. Leaving it open to see if we can have better default resolution once YARN-673 gets resolved. I think we could check the hostname property as well as the rm address property and pick one of those if only one of them is set.
        Hide
        Dave Disser added a comment -

        I'm seeing this issue regardless of the status of yarn.resourcemanager.hostname and yarn.web-proxy.address. I notice the following in my nodemanager log file:

        2014-05-28 14:06:20,478 INFO webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(330)) - dr.who is accessing
        unchecked http://hdp003-3:59959/ which is the app master GUI of application_1401300304842_0001 owned by hdfs

        I can try to retrieve this URL directly:

        hdp003-2:~ # wget -O - http://hdp003-3:59959/
        -2014-05-28 14:06:47- http://hdp003-3:59959/
        Resolving hdp003-3... 39.64.24.3
        Connecting to hdp003-3|39.64.24.3|:59959... connected.
        HTTP request sent, awaiting response... 302 Found
        Location: http://hdp003-3:59959/mapreduce [following]
        -2014-05-28 14:06:47- http://hdp003-3:59959/mapreduce
        Reusing existing connection to hdp003-3:59959.
        HTTP request sent, awaiting response... 302 Found
        Location: http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce [following]
        -2014-05-28 14:06:47- http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce
        Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused.
        Resolving hdp003-3... 39.64.24.3
        Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused.

        The node running the AM is proxying the request to itself, where there is no proxy running. If I do the same on the node where AM is running, I get the proper result:

        hdp003-3:~ # wget -O - http://hdp003-3:59959/
        -2014-05-28 14:07:25- http://hdp003-3:59959/
        Resolving hdp003-3... 39.64.24.3
        Connecting to hdp003-3|39.64.24.3|:59959... connected.
        HTTP request sent, awaiting response... 302 Found
        Location: http://hdp003-3:59959/mapreduce [following]
        -2014-05-28 14:07:25- http://hdp003-3:59959/mapreduce
        Reusing existing connection to hdp003-3:59959.
        HTTP request sent, awaiting response... 200 OK
        Length: 6224 (6.1K) [text/html]
        Saving to: `STDOUT'

        0% [ ] 0 --.-K/s <
        !DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
        <html>
        <meta http-equiv="Content-type" content="text/html; charset=UTF-8">
        <style type="text/css">
        ...

        Show
        Dave Disser added a comment - I'm seeing this issue regardless of the status of yarn.resourcemanager.hostname and yarn.web-proxy.address. I notice the following in my nodemanager log file: 2014-05-28 14:06:20,478 INFO webproxy.WebAppProxyServlet (WebAppProxyServlet.java:doGet(330)) - dr.who is accessing unchecked http://hdp003-3:59959/ which is the app master GUI of application_1401300304842_0001 owned by hdfs I can try to retrieve this URL directly: hdp003-2:~ # wget -O - http://hdp003-3:59959/ - 2014-05-28 14:06:47 - http://hdp003-3:59959/ Resolving hdp003-3... 39.64.24.3 Connecting to hdp003-3|39.64.24.3|:59959... connected. HTTP request sent, awaiting response... 302 Found Location: http://hdp003-3:59959/mapreduce [following] - 2014-05-28 14:06:47 - http://hdp003-3:59959/mapreduce Reusing existing connection to hdp003-3:59959. HTTP request sent, awaiting response... 302 Found Location: http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce [following] - 2014-05-28 14:06:47 - http://hdp003-3:8088/proxy/application_1401300304842_0001/mapreduce Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused. Resolving hdp003-3... 39.64.24.3 Connecting to hdp003-3|39.64.24.3|:8088... failed: Connection refused. The node running the AM is proxying the request to itself, where there is no proxy running. If I do the same on the node where AM is running, I get the proper result: hdp003-3:~ # wget -O - http://hdp003-3:59959/ - 2014-05-28 14:07:25 - http://hdp003-3:59959/ Resolving hdp003-3... 39.64.24.3 Connecting to hdp003-3|39.64.24.3|:59959... connected. HTTP request sent, awaiting response... 302 Found Location: http://hdp003-3:59959/mapreduce [following] - 2014-05-28 14:07:25 - http://hdp003-3:59959/mapreduce Reusing existing connection to hdp003-3:59959. HTTP request sent, awaiting response... 200 OK Length: 6224 (6.1K) [text/html] Saving to: `STDOUT' 0% [ ] 0 --.-K/s < !DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <meta http-equiv="Content-type" content="text/html; charset=UTF-8"> <style type="text/css"> ...
        Hide
        Dave Disser added a comment -

        As a follow-up, I also notice that the proxying works correctly while the tracking URL is "UNASSIGNED" (the first couple seconds after AM container launch), but then HTTP 500 occurs shortly after.

        Show
        Dave Disser added a comment - As a follow-up, I also notice that the proxying works correctly while the tracking URL is "UNASSIGNED" (the first couple seconds after AM container launch), but then HTTP 500 occurs shortly after.

          People

          • Assignee:
            Unassigned
            Reporter:
            Arpit Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development