[HBASE-23917] [SimpleRpcServer] Subsequent requests will have no response in case of request IO mess up

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: rpc
• Assignee: chenxu
• Reporter: chenxu
• Created: 01/Mar/20 19:35
• Updated: 18/Aug/21 16:40

Description

Code in SimpleServerRpcConnection#readAndProcess work like this

public int readAndProcess() throws IOException, InterruptedException {
  …
  if (data == null) {
    …
    initByteBuffToReadInto(dataLength);
    …
  }
  count = channelDataRead(channel, data);
  if (count >= 0 && data.remaining() == 0) { // count==0 if dataLength == 0
    process();
  }
  return count;
}

In case of request IO mess up, data.remaining() may be greater than 0, so process() method will not be executed.

There are some cleanup operations in process()

private void process() throws IOException, InterruptedException {
  data.rewind();
  try {
    ..
  } finally {
    dataLengthBuffer.clear(); // Clean for the next call
    data = null; // For the GC
    this.callCleanup = null;
  }
}

If process() not executed, variable data will always not null, and data.remaining() will always be greater than 0, so process() will never be executed again, and subsequent requests will have no response, this has been occured in our product env.

Issue Links

• links to: GitHub Pull Request #1231