Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8452

DN logs backtrace when running under jsvc and /jmx is loaded

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.0.2-alpha
    • None
    • None

    Description

      Running the data node under jsvc and requesting /jmx falls victim to a kernel bug http://marc.info/?l=linux-kernel&m=133788505209725&w=2 which results in EACCES when open()ing /proc/self/fd to attempt to count the open filedescriptors.

      Hopefully someday we will have kernels with this bug fixed; in the meantime, the log spew which results is unpleasant:

      12270 2012-05-15 21:04:41,683 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute OpenFileDescriptorCount of java.lang:type=OperatingSystem threw an exception
      12271 javax.management.RuntimeErrorException: java.lang.InternalError: errno: 13 error: Unable to open directory /proc/self/fd
      12272
      12273 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
      12274 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
      12275 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
      12276 at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
      12277 at org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:314)
      12278 at org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:292)
      12279 at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:192)
      12280 at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      12281 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      12282 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      12283 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      12284 at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
      12285 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      12286 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      12287 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      12288 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      12289 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      12290 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      12291 at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      12292 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      12293 at org.mortbay.jetty.Server.handle(Server.java:326)
      12294 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      12295 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      12296 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      12297 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      12298 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      12299 at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      12300 at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      12301 Caused by: java.lang.InternalError: errno: 13 error: Unable to open directory /proc/self/fd
      12302
      12303 at com.sun.management.UnixOperatingSystem.getOpenFileDescriptorCount(Native Method)
      12304 at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
      12305 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      12306 at java.lang.reflect.Method.invoke(Method.java:597)
      12307 at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
      12308 at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
      12309 at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
      12310 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
      12311 at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
      12312 at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
      12313 at javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
      12314 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
      12315 ... 25 more

      we should catch the RuntimeErrorException and make it a debug().

      Attachments

        1. 95db1fa.txt
          1 kB
          Andy Isaacson
        2. hadoop-8447.patch
          1 kB
          Andy Isaacson
        3. hadoop-8447-2.patch
          2 kB
          Andy Isaacson
        4. hadoop-8447-2.patch
          2 kB
          Andy Isaacson

        Activity

          People

            adi2 Andy Isaacson
            adi2 Andy Isaacson
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: