[HBASE-12025] TestHttpServerLifecycle.testStartedServerWithRequestLog hangs frequently

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.99.1
• Component/s: test
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 19/Sep/14 04:02
• Updated: 06/Apr/18 17:55
• Resolved: 19/Sep/14 04:09

Description

Happened over on https://builds.apache.org/job/PreCommit-HBASE-Build/10974/consoleText most recently. The test is to see if can enable request logging in jetty (we should remove it?). Looks like some bug in our ancient jetty version. Let me for now change the stop order so the request listener is removed before we go to shut down. Here is the bind:

"1738904334@qtp-1379460953-1 - Acceptor0 SelectChannelConnector@localhost:56958" daemon prio=10 tid=0x00007f493884b800 nid=0x14b9 waiting for monitor entry [0x00007f490a59e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:464)
	- waiting to lock <0x00000007dd6a1028> (a org.mortbay.io.nio.SelectorManager$SelectSet)
	at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
	at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
	at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"825895443@qtp-1379460953-0" daemon prio=10 tid=0x00007f493885a000 nid=0x14b8 in Object.wait() [0x00007f490a49d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007dcd02220> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
	- locked <0x00000007dcd02220> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

"Service Thread" daemon prio=10 tid=0x00007f4938290000 nid=0x14ab runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f493828e000 nid=0x14aa waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f493828b000 nid=0x14a9 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f4938280800 nid=0x14a8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f493826b000 nid=0x14a7 in Object.wait() [0x00007f490b3f2000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007d8685568> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
	- locked <0x00000007d8685568> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)

"Reference Handler" daemon prio=10 tid=0x00007f4938267000 nid=0x14a6 in Object.wait() [0x00007f490b4f3000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007d86850f0> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:503)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
	- locked <0x00000007d86850f0> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f493800a800 nid=0x1497 runnable [0x00007f4941b5f000]
   java.lang.Thread.State: RUNNABLE
	at org.mortbay.io.nio.SelectorManager$SelectSet.stop(SelectorManager.java:879)
	- locked <0x00000007dd6a1028> (a org.mortbay.io.nio.SelectorManager$SelectSet)
	at org.mortbay.io.nio.SelectorManager.doStop(SelectorManager.java:240)
	at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
	- locked <0x00000007dccf2e60> (a java.lang.Object)
	at org.mortbay.jetty.nio.SelectChannelConnector.close(SelectChannelConnector.java:136)
	- locked <0x00000007dccf2cc0> (a org.mortbay.jetty.nio.SelectChannelConnector)
	at org.apache.hadoop.hbase.http.HttpServer.stop(HttpServer.java:1037)
	at org.apache.hadoop.hbase.http.HttpServerFunctionalTest.stop(HttpServerFunctionalTest.java:195)
	at org.apache.hadoop.hbase.http.TestHttpServerLifecycle.testStartedServerWithRequestLog(TestHttpServerLifecycle.java:91)

Issue Links

• is related to: HBASE-12158 TestHttpServerLifecycle.testStartedServerWithRequestLog goes zombie on occasion