Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-12055

TestBucketWriterThread hangs flakily based on timing

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.7, 0.99.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestBucketWriterThread sometimes hangs (esp on windows) due to timing issues with regards to the thread start and blocking queue start.

      Test output:

      -------------------------------------------------------
       T E S T S
      -------------------------------------------------------
      Running org.apache.hadoop.hbase.io.hfile.bucket.TestBucketWriterThread
      2014-09-17 13:58:43,607 INFO  [main] hbase.ResourceChecker(147): before: io.hfile.bucket.TestBucketWriterThread#testNonE
      rrorCase Thread=5, OpenFileDescriptor=0, MaxFileDescriptor=0, SystemLoadAverage=0, ProcessCount=0, AvailableMemoryMB=0,
      ConnectionCount=0
      2014-09-17 13:58:43,625 INFO  [main] util.ByteBufferArray(57): Allocating buffers total=16 B , sizePerBuffer=32 KB, coun
      t=1
      2014-09-17 13:58:43,631 INFO  [main] bucket.BucketCache(271): Started bucket cache; ioengine=heap, capacity=16 B, blockS
      ize=1 B, writerThreadNum=1, writerQLen=16, persistencePath=null, bucketAllocator=bucket.0: size=1, freeCount=4, used=0,
      bucket.1: size=1, freeCount=4, used=0, bucket.2: size=1, freeCount=4, used=0, bucket.3: size=1, freeCount=4, used=0
      2014-09-17 13:58:43,716 INFO  [main-BucketCacheWriter-0] bucket.BucketCache$WriterThread(727): main-BucketCacheWriter-0
      exiting, cacheEnabled=true
      2014-09-17 14:01:44,024 DEBUG [BucketCache Statistics #0] bucket.BucketCache(497): BucketCache Stats: failedBlockAdditio
      ns=0, total=16 B, free=16 B, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRa
      tio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
      2014-09-17 14:04:44,206 DEBUG [BucketCache Statistics #0] bucket.BucketCache(497): BucketCache Stats: failedBlockAdditio
      ns=0, total=16 B, free=16 B, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRa
      tio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN
      

      Stack dump:

      2014-09-17 14:01:51
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):
      
      "BucketCache Statistics #0" daemon prio=6 tid=0x0000000008293000 nid=0x7e8 waiting on condition [0x000000000712f000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x000000077ad10110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
      	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Low Memory Detector" daemon prio=6 tid=0x0000000000526000 nid=0x176c runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "C2 CompilerThread1" daemon prio=10 tid=0x0000000000524800 nid=0x428 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "C2 CompilerThread0" daemon prio=10 tid=0x0000000000513000 nid=0x5d0 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Attach Listener" daemon prio=10 tid=0x0000000000511000 nid=0x944 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Signal Dispatcher" daemon prio=10 tid=0x000000000050e000 nid=0x870 runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Finalizer" daemon prio=8 tid=0x00000000004f5800 nid=0x460 in Object.wait() [0x000000000602f000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x000000077ad104c8> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      	- locked <0x000000077ad104c8> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
      	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      
      "Reference Handler" daemon prio=10 tid=0x00000000004f3800 nid=0x67c in Object.wait() [0x0000000005f2f000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x000000077ad10568> (a java.lang.ref.Reference$Lock)
      	at java.lang.Object.wait(Object.java:485)
      	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      	- locked <0x000000077ad10568> (a java.lang.ref.Reference$Lock)
      
      "main" prio=6 tid=0x000000000025d800 nid=0xb24 waiting on condition [0x00000000025ae000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:143)
      	at org.apache.hadoop.hbase.io.hfile.bucket.TestBucketWriterThread.setUp(TestBucketWriterThread.java:78)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
      	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
      	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
      
      "VM Thread" prio=10 tid=0x00000000004ed000 nid=0x550 runnable 
      
      "VM Periodic Task Thread" prio=10 tid=0x000000000052d800 nid=0xafc waiting on condition 
      
      JNI global references: 1395
      
      

        Attachments

        1. hbase-12055_v1.patch
          5 kB
          Enis Soztutar

          Issue Links

            Activity

              People

              • Assignee:
                ndimiduk Nick Dimiduk
                Reporter:
                enis Enis Soztutar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: