[HADOOP-8452] DN logs backtrace when running under jsvc and /jmx is loaded

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.0.2-alpha
• Assignee: Andy Isaacson
• Reporter: Andy Isaacson
• Created: 30/May/12 01:24
• Updated: 04/Sep/14 00:59
• Resolved: 31/May/12 14:12

Description

Running the data node under jsvc and requesting /jmx falls victim to a kernel bug http://marc.info/?l=linux-kernel&m=133788505209725&w=2 which results in EACCES when open()ing /proc/self/fd to attempt to count the open filedescriptors.

Hopefully someday we will have kernels with this bug fixed; in the meantime, the log spew which results is unpleasant:

12270 2012-05-15 21:04:41,683 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute OpenFileDescriptorCount of java.lang:type=OperatingSystem threw an exception
12271 javax.management.RuntimeErrorException: java.lang.InternalError: errno: 13 error: Unable to open directory /proc/self/fd
12272
12273 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
12274 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
12275 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
12276 at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
12277 at org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:314)
12278 at org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:292)
12279 at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:192)
12280 at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
12281 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
12282 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
12283 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
12284 at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
12285 at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
12286 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
12287 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
12288 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
12289 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
12290 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
12291 at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
12292 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
12293 at org.mortbay.jetty.Server.handle(Server.java:326)
12294 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
12295 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
12296 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
12297 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
12298 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
12299 at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
12300 at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
12301 Caused by: java.lang.InternalError: errno: 13 error: Unable to open directory /proc/self/fd
12302
12303 at com.sun.management.UnixOperatingSystem.getOpenFileDescriptorCount(Native Method)
12304 at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
12305 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
12306 at java.lang.reflect.Method.invoke(Method.java:597)
12307 at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
12308 at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
12309 at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
12310 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
12311 at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
12312 at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
12313 at javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
12314 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
12315 ... 25 more

we should catch the RuntimeErrorException and make it a debug().