Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6031

RegionServer does not go down while aborting

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.98.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Fixed in hadoop 2.0.3-alpha.

      Description

      Following is the thread dump.

      "1997531088@qtp-716941846-5" prio=10 tid=0x00007f7c5820c800 nid=0xe1b in Object.wait() [0x00007f7c56ae8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
      	- locked <0x00007f7cfe0616d0> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
      	at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
      	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
      	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
      	at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
      	- locked <0x00007f7cfe74d758> (a org.mortbay.util.ByteArrayOutputStream2)
      	at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
      	at java.io.Writer.write(Writer.java:96)
      	- locked <0x00007f7cfca02fc0> (a org.mortbay.jetty.HttpConnection$OutputWriter)
      	at java.io.PrintWriter.write(PrintWriter.java:361)
      	- locked <0x00007f7cfca02fc0> (a org.mortbay.jetty.HttpConnection$OutputWriter)
      	at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
      	at org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
      	at org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      	at org.mortbay.jetty.Server.handle(Server.java:326)
      	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      
      "1374615312@qtp-716941846-3" prio=10 tid=0x00007f7c58214800 nid=0xc42 in Object.wait() [0x00007f7c55bd9000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
      	- locked <0x00007f7cfdbb6cc8> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
      	at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
      	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
      	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
      	at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
      	- locked <0x00007f7cfe7dbda0> (a org.mortbay.util.ByteArrayOutputStream2)
      	at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
      	at java.io.Writer.write(Writer.java:96)
      	- locked <0x00007f7cfe30ebe0> (a org.mortbay.jetty.HttpConnection$OutputWriter)
      	at java.io.PrintWriter.write(PrintWriter.java:361)
      	- locked <0x00007f7cfe30ebe0> (a org.mortbay.jetty.HttpConnection$OutputWriter)
      	at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
      	at org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
      	at org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      	at org.mortbay.jetty.Server.handle(Server.java:326)
      	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      
      "615294984@qtp-716941846-2" prio=10 tid=0x00007f7c58086000 nid=0x907 in Object.wait() [0x00007f7c57ffd000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:279)
      	- locked <0x00007f7d6383cd40> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)
      	at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:545)
      	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:639)
      	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
      	at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
      	- locked <0x00007f7d116bbf90> (a org.mortbay.util.ByteArrayOutputStream2)
      	at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:904)
      	at java.io.Writer.write(Writer.java:96)
      	- locked <0x00007f7d54b19ea0> (a org.mortbay.jetty.HttpConnection$OutputWriter)
      	at java.io.PrintWriter.write(PrintWriter.java:361)
      	- locked <0x00007f7d54b19ea0> (a org.mortbay.jetty.HttpConnection$OutputWriter)
      	at org.jamon.escaping.HtmlEscaping.write(HtmlEscaping.java:43)
      	at org.jamon.escaping.AbstractCharacterEscaping.write(AbstractCharacterEscaping.java:35)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmplImpl.renderNoFlush(RSStatusTmplImpl.java:222)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.renderNoFlush(RSStatusTmpl.java:180)
      	at org.apache.hadoop.hbase.tmpl.regionserver.RSStatusTmpl.render(RSStatusTmpl.java:171)
      	at org.apache.hadoop.hbase.regionserver.RSStatusServlet.doGet(RSStatusServlet.java:48)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
      	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      	at org.mortbay.jetty.Server.handle(Server.java:326)
      	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      
      "regionserver60020.decayingSampleTick.1" daemon prio=10 tid=0x0000000040b2b800 nid=0x6f1a waiting on condition [0x00007f7c697a5000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00007f7c9eb40cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
      	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:619)
      
      "LeaseRenewer:root@10.18.40.217:9000" daemon prio=10 tid=0x00007f7c6c770000 nid=0x6f18 waiting on condition [0x00007f7c699a7000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:432)
      	at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:69)
      	at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:299)
      	at java.lang.Thread.run(Thread.java:619)
      

        Attachments

        1. rs_shutdown_hung20130107.jstack
          10 kB
          Liang Xie
        2. rsthread.txt
          184 kB
          ramkrishna.s.vasudevan

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ram_krish ramkrishna.s.vasudevan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: