[HDFS-6478] RemoteException can't be retried properly for non-HA scenario

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 02/Jun/14 23:25
• Updated: 27/Apr/15 18:39
• Resolved: 17/Jul/14 17:40

Description

For HA case, the call stack is DFSClient -> RetryInvocationHandler -> ClientNamenodeProtocolTranslatorPB -> ProtobufRpcEngine. ProtobufRpcEngine. ProtobufRpcEngine throws ServiceException and expects the caller to unwrap it; ClientNamenodeProtocolTranslatorPB is the component that takes care of that.

        at org.apache.hadoop.ipc.Client.call
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke
        at com.sun.proxy.$Proxy26.getFileInfo
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo
        at sun.reflect.GeneratedMethodAccessor24.invoke
        at sun.reflect.DelegatingMethodAccessorImpl.invoke
        at java.lang.reflect.Method.invoke
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke
        at com.sun.proxy.$Proxy27.getFileInfo
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus

However, for non-HA case, the call stack is DFSClient -> ClientNamenodeProtocolTranslatorPB -> RetryInvocationHandler -> ProtobufRpcEngine. RetryInvocationHandler gets ServiceException and can't be retried properly.

at org.apache.hadoop.ipc.Client.call
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke
at com.sun.proxy.$Proxy9.getListing
at sun.reflect.NativeMethodAccessorImpl.invoke0
at sun.reflect.NativeMethodAccessorImpl.invoke
at sun.reflect.DelegatingMethodAccessorImpl.invoke
at java.lang.reflect.Method.invoke
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke
at com.sun.proxy.$Proxy9.getListing
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing
at org.apache.hadoop.hdfs.DFSClient.listPaths

Perhaps, we can fix it by have NN wrap RetryInvocationHandler around ClientNamenodeProtocolTranslatorPB and other PBs, instead of the current wrap order.

Issue Links

• breaks: HBASE-13574 Broken TestHBaseFsck in master with hadoop 2.6.0
• breaks: HBASE-12607 TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0