Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-13851

RpcClientImpl.close() can hang with cancelled replica RPCs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.2, 1.2.0, 1.1.1, 2.0.0
    • Component/s: IPC/RPC
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We have seen the clients hanging in running the test IntegrationTestRegionReplicaPerf in 1.1 code base during the test.The jstack gives:

      "IPC Client (1344340481) connection to os-enis-dal-test-jun-4-1.openstacklocal/172.22.80.25:16020 from root - writer" daemon prio=10 tid=0x00007f3891b29800 nid=0x7345 waiting on condition [0x00007f3865647000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x000000070d54a240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
              at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
              at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$CallSender.run(RpcClientImpl.java:253)
      
      "TestClient-3" prio=10 tid=0x00007f3892660800 nid=0x63b0 waiting on condition [0x00007f386ecdd000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.hadoop.hbase.ipc.RpcClientImpl.close(RpcClientImpl.java:1139)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.internalClose(ConnectionManager.java:2371)
              at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.close(ConnectionManager.java:2384)
              at org.apache.hadoop.hbase.PerformanceEvaluation$Test.testTakedown(PerformanceEvaluation.java:1036)
              at org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testTakedown(PerformanceEvaluation.java:1351)
              at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1055)
              at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1612)
              at org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:410)
              at org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:405)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. hbase-13851_v1.patch
          1.0 kB
          Enis Soztutar

          Activity

            People

            • Assignee:
              enis Enis Soztutar
              Reporter:
              enis Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: