[HBASE-9129] Scanner prefetching leaks scanners.

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.95.1
• Fix Version/s: 0.95.2
• Hadoop Flags: Reviewed
• Release Note: Reverted Scanner Pre-fetching
• Assignee: Elliott Neil Clark
• Reporter: Elliott Neil Clark
• Created: 05/Aug/13 22:33
• Updated: 23/Sep/13 19:22
• Resolved: 06/Aug/13 20:11

Description

Running YCSB workload-e on 0.95 basically hangs the whole cluster.

YCSB opens a scanner for 100 rows.

1. YCSB Calls next
2. Pre-fetching starts
3. YCSB Closes scanner
4. Pre-fetching re-adds the scanner.

So the end result is:

"scan-prefetch-2-thread-45" daemon prio=10 tid=0x00007f7e406ec800 nid=0x40bc runnable [0x00007f75ffefd000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272)
        at java.util.PriorityQueue.indexOf(PriorityQueue.java:342)
        at java.util.PriorityQueue.remove(PriorityQueue.java:360)
        at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476)
        at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232)
        - locked <0x00007f774455a660> (a java.util.concurrent.DelayQueue)
        at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269)
        at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

"scan-prefetch-2-thread-44" daemon prio=10 tid=0x00007f7e4c1ba800 nid=0x40bb waiting on condition [0x00007f7605b79000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f774455ad78> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

"scan-prefetch-2-thread-43" daemon prio=10 tid=0x00007f7e38cbc800 nid=0x40ba waiting on condition [0x00007f7609ab8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f774455ad78> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)