Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14198

Should have a way to let PingInputStream to abort

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We observed a case that RPC call get stuck, since PingInputStream does the following

       /** This class sends a ping to the remote side when timeout on
           * reading. If no failure is detected, it retries until at least
           * a byte is read.
           */
          private class PingInputStream extends FilterInputStream {
      

      It seems that in this case no data is ever received, and it keeps pinging.

      Should we ping forever here? Maybe we should introduce a config to stop the ping after pinging for certain number of times, and report back timeout, let the caller to retry the RPC?

      Wonder if there is chance the RPC get dropped somehow by the server so no response is ever received.

      See

      Thread 16127: (state = BLOCKED)                                                                     
       - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled frame)                   
       - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 (Compiled frame)       
       - org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) @bci=5, line=57 (Compiled frame)
       - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=35, line=142 (Compiled frame)
       - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=161 (Compiled frame)
       - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=131 (Compiled frame) 
       - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame)               
       - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame)               
       - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=521 (Compiled frame)
       - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)                           
       - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)                            
       - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)                              
       - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, line=1081 (Compiled frame) 
       - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled frame)         
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                yzhangal Yongjun Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: