[TAJO-1563] Improve RPC error handling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: QueryMaster, RPC, TajoMaster, Worker
• Assignee: Jinho Kim
• Reporter: Jinho Kim
• Created: 17/Apr/15 06:59
• Updated: 02/Sep/15 05:23
• Resolved: 28/Apr/15 09:29

Description

Currently, RPC does not handle error for disconnection and a client does not throw if async RPC failed.
for example. when a client calling and then it has lost a connection, callback has not been called.
Callfuture.get() should throw ExecutionException and RPC client should handle a recoverable exception

• Server internal exception handling
  • Application exceptions
    • send RemoteCallException to client
  • Unexpected exceptions (Server failure)
    • close current channel (client will receive channel inactive event)

• Client internal exception handling
  • Add internal retry when a stub is failed by connection lost
  • Application exceptions
    • throw exception to callback
  • Max retry exceeded
    • throw RecoverableException to internal exception handler
  • Server hangs(server shutdown, OOM..)
    • trigger the ping expired event. and throw ServiceException
  • Unexpected exceptions (Client failure, connection lost)
    • throw exception to all callback

Issue Links

• contains: TAJO-1540 RpcCallback must be able to handle TimeoutException or cancel.
• is related to: TAJO-1569 BlockingRpcClient can make other request fail
• is related to: TAJO-1214 (Umbrella) Improve task and node failure handling
• is required by: TAJO-1584 Remove QueryMaster client sharing in TajoMaster and TajoWorker
• is required by: TAJO-1583 Remove ServerCallable in RPC client