[HADOOP-16268] Allow custom wrapped exception to be thrown by server if RPC call queue is filled up

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Assignee: CR Hota
• Reporter: CR Hota
• Created: 22/Apr/19 22:04
• Updated: 02/Oct/19 17:14
• Resolved: 04/Sep/19 15:22

Description

In the current implementation of callqueue manager, "CallQueueOverflowException" exceptions are always wrapping "RetriableException". Through configs servers should be allowed to throw custom exceptions based on new use cases.

In CallQueueManager.java for backoff the below is done 

  // ideally this behavior should be controllable too.
  private void throwBackoff() throws IllegalStateException {
    throw CallQueueOverflowException.DISCONNECT;
  }

Since CallQueueOverflowException only wraps RetriableException clients would end up hitting the same server for retries. In use cases that router supports these overflowed requests could be handled by another router that shares the same state thus distributing load across a cluster of routers better. In the absence of any custom exception, current behavior should be supported.

In CallQueueOverflowException class a new Standby exception wrap should be created. Something like the below

   static final CallQueueOverflowException KEEPALIVE =
        new CallQueueOverflowException(
            new RetriableException(TOO_BUSY),
            RpcStatusProto.ERROR);
    static final CallQueueOverflowException DISCONNECT =
        new CallQueueOverflowException(
            new RetriableException(TOO_BUSY + " - disconnecting"),
            RpcStatusProto.FATAL);
    static final CallQueueOverflowException DISCONNECT2 =
        new CallQueueOverflowException(
            new StandbyException(TOO_BUSY + " - disconnecting"),
            RpcStatusProto.FATAL);