Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15140

Fix ResultBoundedCompletionService deadlock

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 1.1.2
    • Fix Version/s: 1.1.4
    • Component/s: Client
    • Labels:
      None

      Description

      See: https://issues.apache.org/jira/browse/HBASE-14812

      We ran into this deadlock issue on the hbase 1.1.2 build

      phoenix-1-thread-1340 id=3183 state=WAITING
          - waiting on <0x1ad981d3> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          - locked <0x1ad981d3> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
          at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
          at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
          at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
          at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
          at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
          at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
      
      phoenix-1-thread-1341 id=3184 state=WAITING
          - waiting on <0x22e46b2c> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          - locked <0x22e46b2c> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
          at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
          at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
          at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
          at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
          at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
          at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@25efbbe8
      
      phoenix-1-thread-1342 id=3185 state=WAITING
          - waiting on <0x6723add5> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          - locked <0x6723add5> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
          at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
          at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
          at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
          at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
          at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
          at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@8561ef4
      
      phoenix-1-thread-1343 id=3186 state=WAITING
          - waiting on <0x31fd0245> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          - locked <0x31fd0245> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
          at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
          at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
          at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
          at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
          at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
          at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@13961870
      
      phoenix-1-thread-1344 id=3187 state=WAITING
          - waiting on <0x79f31d27> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          - locked <0x79f31d27> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
          at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
          at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
          at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
          at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
          at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
          at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
          at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@64d45bc2
      
      phoenix-1-thread-1345 id=3188 state=WAITING
          - waiting on <0x331112aa> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          - locked <0x331112aa> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
          at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
          at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
          at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
          at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
          at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
          at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
          at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
          at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
          at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
      

      The patch attached to HBASE-14812 applies cleanly to version 1.1.2, so, it should easy to port and apply.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                alok_at_cloudability Alok Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: