[HBASE-7684] NullPointerException in SecureClient when Call is cleaned up due to RPC timeout

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.94.3
• Fix Version/s: 0.94.5
• Component/s: security
• Assignee: Jianwei Cui
• Reporter: Jianwei Cui
• Created: 27/Jan/13 09:06
• Updated: 16/Feb/13 04:14
• Resolved: 30/Jan/13 05:43

Description

We find NullPointException when using SecureClient to access hbase. From the source code, we find that receiveResponse() will not check call != null before invoke call.setValue(...)(line 378 and 380 in SecureClient.java). However, as explained in receiveResponse() of HBaseClient, the call may have been cleaned up due to RPC timeout; therefore, it should be better to check call != null before invoke call.setValue(...) in SecureClient.
The exception stack trace is :

  2013-01-28 12:11:20,060 [request-queuepool-10-thread-1] WARN  org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Failed all from region=sms_message,98,1358995149987.0c18f72df3f3f398f9f6dc83fe65afad., hostname=10.20.2.72, port=11600
    2013-01-28 12:11:20,060 [request-queuepool-10-thread-1] WARN  org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Failed all from region=sms_message,98,1358995149987.0c18f72df3f3f398f9f6dc83fe65afad., hostname=10.20.2.72, port=11600
    java.util.concurrent.ExecutionException: java.io.IOException: Call to 10.20.2.72/10.20.2.72:11600 failed on local exception: java.io.IOException: Unexpected exception receiving call responses
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
      at java.util.concurrent.FutureTask.get(FutureTask.java:83)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1544)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1396)
      at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:937)
      at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
      at org.apache.hadoop.hbase.client.HTable.put(HTable.java:752)
      at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:397)
      at com.xiaomi.infra.hbase.business.dao.sms.SmsMessage.updateStatus(SmsMessage.java:245)
      at com.xiaomi.infra.hbase.business.client.sms.HMessageDao.doUpdateMessage(HMessageDao.java:165)
      at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.xiaomi.infra.hbase.business.client.AsyncDaoWrapper.wrapperRunMethod(AsyncDaoWrapper.java:97)
      at com.xiaomi.infra.hbase.business.client.HBaseRequest.run(AsyncDaoWrapper.java:135)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Call to 10.20.2.72/10.20.2.72:11600 failed on local exception: java.io.IOException: Unexpected exception receiving call responses
      at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1056)
      at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1025)
      at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:165)
      at $Proxy7.multi(Unknown Source)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1373)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1)
      at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:210)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1380)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      ... 3 more
    Caused by: java.io.IOException: Unexpected exception receiving call responses
      at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:610)
    Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.receiveResponse(SecureClient.java:378)
      at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:606)