[IMPALA-2908] Find a permanent fix to avoid a cluster hang / incorrect results on a failed EOS RPC

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: Impala 2.3.0
• Component/s: Distributed Exec
• Labels: correctness hang usability
• Target Version: Impala 2.6.0
• Assignee: Henry Robinson
• Reporter: Sailesh Mukil
• Created: 29/Jan/16 07:51
• Updated: 31/Mar/16 18:19
• Resolved: 31/Mar/16 18:19

Description

In DataStreamSender::Channel::CloseInternal(), if the EOS RPC fails, the receiver side of the RPC will remain open indefinitely causing the cluster to hang. If the sending of the last row-batch fails during CloseInternal(), then the query can succeed but return incorrect results. Only an error would be logged via LogError() in this case.

This is an incremental task after IMPALA-2592 which was a temporary fix to reduce the window of vulnerability. However, this needs a more well thought of permanent fix.