Hadoop YARN
  1. Hadoop YARN
  2. YARN-800

Clicking on an AM link for a running app leads to a HTTP 500

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Clicking the AM link tries to open up a page with url like

      http://hostname:8088/proxy/application_1370886527995_0645/

      and this leads to an HTTP 500

        Activity

        Hide
        Arpit Gupta added a comment -

        Changing it to minor. If i set yarn.resourcemanager.hostname webapp adress default gets picked up correctly. Leaving it open to see if we can have better default resolution once YARN-673 gets resolved. I think we could check the hostname property as well as the rm address property and pick one of those if only one of them is set.

        Show
        Arpit Gupta added a comment - Changing it to minor. If i set yarn.resourcemanager.hostname webapp adress default gets picked up correctly. Leaving it open to see if we can have better default resolution once YARN-673 gets resolved. I think we could check the hostname property as well as the rm address property and pick one of those if only one of them is set.
        Hide
        Jian He added a comment -

        The link is not broken in local cluster because by default local cluster use localhost(0.0.0.0) as RM Ip Address.
        This problem occurs on real cluster where RM is using real IP address, and RM_WEBAPP_ADDRESS is still defaulting to '0.0.0.0', we should default RM_WEBAPP_ADDRESS to RM_ADDRESS.
        If we specify RM_WEBAPP_ADDRESS explicitly, this problem will not occur

        Show
        Jian He added a comment - The link is not broken in local cluster because by default local cluster use localhost(0.0.0.0) as RM Ip Address. This problem occurs on real cluster where RM is using real IP address, and RM_WEBAPP_ADDRESS is still defaulting to '0.0.0.0', we should default RM_WEBAPP_ADDRESS to RM_ADDRESS. If we specify RM_WEBAPP_ADDRESS explicitly, this problem will not occur
        Hide
        Zhijie Shen added a comment -

        Did a quick local test, and found the link was not broken. It seems that the default value has already been in yarn-default.xml

          <property>
            <description>The hostname of the RM.</description>
            <name>yarn.resourcemanager.hostname</name>
            <value>0.0.0.0</value>
          </property> 
        
          <property>
            <description>The address of the RM web application.</description>
            <name>yarn.resourcemanager.webapp.address</name>
            <value>${yarn.resourcemanager.hostname}:8088</value>
          </property>
        

        and YarnConfiguration

          public static final String RM_WEBAPP_ADDRESS = 
            RM_PREFIX + "webapp.address";
        
          public static final int DEFAULT_RM_WEBAPP_PORT = 8088;
          public static final String DEFAULT_RM_WEBAPP_ADDRESS = "0.0.0.0:" +
            DEFAULT_RM_WEBAPP_PORT;
        

        Looked into the code, it seems to be related to "yarn.web-proxy.address". In WebAppProxyServlet,

              resp.setStatus(client.executeMethod(config, method));
        

        tries to connect the proxy host to show the application webpage. If "yarn.web-proxy.address" is not set, RM will become the proxy, and its address will be ${yarn.resourcemanager.hostname}:8088 as well.

        Maybe it is good to check the configuration of "yarn.web-proxy.address"

        Show
        Zhijie Shen added a comment - Did a quick local test, and found the link was not broken. It seems that the default value has already been in yarn-default.xml <property> <description>The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>0.0.0.0</value> </property> <property> <description>The address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property> and YarnConfiguration public static final String RM_WEBAPP_ADDRESS = RM_PREFIX + "webapp.address" ; public static final int DEFAULT_RM_WEBAPP_PORT = 8088; public static final String DEFAULT_RM_WEBAPP_ADDRESS = "0.0.0.0:" + DEFAULT_RM_WEBAPP_PORT; Looked into the code, it seems to be related to "yarn.web-proxy.address". In WebAppProxyServlet, resp.setStatus(client.executeMethod(config, method)); tries to connect the proxy host to show the application webpage. If "yarn.web-proxy.address" is not set, RM will become the proxy, and its address will be ${yarn.resourcemanager.hostname}:8088 as well. Maybe it is good to check the configuration of "yarn.web-proxy.address"
        Hide
        Arpit Gupta added a comment -

        Looks like we have to set the property yarn.resourcemanager.webapp.address to RMAddress:8088 which should not be the case. We should be defaulting the appropriate value in the system.

        Show
        Arpit Gupta added a comment - Looks like we have to set the property yarn.resourcemanager.webapp.address to RMAddress:8088 which should not be the case. We should be defaulting the appropriate value in the system.
        Hide
        Arpit Gupta added a comment -

        Here is the stack trace

        HTTP ERROR 500
        
        Problem accessing /proxy/application_1370886527995_0658/. Reason:
        
            Connection refused
        
        Caused by:
        
        java.net.ConnectException: Connection refused
        	at java.net.PlainSocketImpl.socketConnect(Native Method)
        	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        	at java.net.Socket.connect(Socket.java:529)
        	at java.net.Socket.connect(Socket.java:478)
        	at java.net.Socket.<init>(Socket.java:375)
        	at java.net.Socket.<init>(Socket.java:249)
        	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
        	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
        	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
        	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
        	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
        	at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185)
        	at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:334)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
        	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
        	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
        	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
        	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1077)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        	at org.mortbay.jetty.Server.handle(Server.java:326)
        	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
        
        Powered by Jetty://
        
        Show
        Arpit Gupta added a comment - Here is the stack trace HTTP ERROR 500 Problem accessing /proxy/application_1370886527995_0658/. Reason: Connection refused Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at java.net.Socket.<init>(Socket.java:375) at java.net.Socket.<init>(Socket.java:249) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1077) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Powered by Jetty: //

          People

          • Assignee:
            Unassigned
            Reporter:
            Arpit Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development