Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6478

RemoteException can't be retried properly for non-HA scenario

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For HA case, the call stack is DFSClient -> RetryInvocationHandler -> ClientNamenodeProtocolTranslatorPB -> ProtobufRpcEngine. ProtobufRpcEngine. ProtobufRpcEngine throws ServiceException and expects the caller to unwrap it; ClientNamenodeProtocolTranslatorPB is the component that takes care of that.

              at org.apache.hadoop.ipc.Client.call
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke
              at com.sun.proxy.$Proxy26.getFileInfo
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo
              at sun.reflect.GeneratedMethodAccessor24.invoke
              at sun.reflect.DelegatingMethodAccessorImpl.invoke
              at java.lang.reflect.Method.invoke
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke
              at com.sun.proxy.$Proxy27.getFileInfo
              at org.apache.hadoop.hdfs.DFSClient.getFileInfo
              at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus
      

      However, for non-HA case, the call stack is DFSClient -> ClientNamenodeProtocolTranslatorPB -> RetryInvocationHandler -> ProtobufRpcEngine. RetryInvocationHandler gets ServiceException and can't be retried properly.

      at org.apache.hadoop.ipc.Client.call
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke
      at com.sun.proxy.$Proxy9.getListing
      at sun.reflect.NativeMethodAccessorImpl.invoke0
      at sun.reflect.NativeMethodAccessorImpl.invoke
      at sun.reflect.DelegatingMethodAccessorImpl.invoke
      at java.lang.reflect.Method.invoke
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke
      at com.sun.proxy.$Proxy9.getListing
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing
      at org.apache.hadoop.hdfs.DFSClient.listPaths
      

      Perhaps, we can fix it by have NN wrap RetryInvocationHandler around ClientNamenodeProtocolTranslatorPB and other PBs, instead of the current wrap order.

        Attachments

        1. HDFS-6478.patch
          18 kB
          Ming Ma
        2. HDFS-6478-2.patch
          20 kB
          Ming Ma
        3. HDFS-6478-3.patch
          14 kB
          Ming Ma
        4. HDFS-6478-4.patch
          15 kB
          Ming Ma

          Issue Links

            Activity

              People

              • Assignee:
                mingma Ming Ma
                Reporter:
                mingma Ming Ma
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: