[IMPALA-3990] ExchangeNode::Close() cancel sender fragment if called before eos

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.7.0
• Component/s: Distributed Exec
• Labels: resource-management
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 17/Aug/16 19:21
• Updated: 19/May/20 20:24

Description

Currently ExchangeNode::Close() does not notify the sender fragment that the stream was closed. This works in a some cases:

• The full input was consumed - since the sender fragment is closed anyway.
• The sender and receiver fragments are both being closed as part of query cleanup.
• ExchangeNode::Close() was called as a result of hitting a limit in the coordinator fragment - the query finished successfully and the coordinator will cancel all fragments immediately.

It doesn't work in another case that would be useful: if only part of the input has been consumed somewhere in the middle of the plan, but the consuming node has consumed all the required input (e.g. short-circuiting join optimisations like IMPALA-3987 or if a limit has been hit). In those cases we could close the exchange node and sending fragment to clean up resources.

What happens if you do this is:

• The ExchangeNode is closed, and the DataStreamRecvr is added to closed_recvrs_cache_
• The sending fragment keeps sending rows, which are discarded when the receiver is found in closed_recvrs_cache_
• Once the receiver ages out of closed_recvrs_cache_, an error will be propagated back to the sender, which will fail the query.

This is bad in that it does a lot of unnecessary processing, holds onto resources, and can cause queries to fail.

We should really notify the sender that the receiver closed down, so that it can close the fragment once all receivers are closed.

Issue Links

• blocks: IMPALA-3987 Short-circuit evaluation of hash joins with empty build side
• is related to: IMPALA-6984 Coordinator should cancel backends when returning EOS
• is related to: IMPALA-8845 Close ExecNode tree prior to calling FlushFinal in FragmentInstanceState
• relates to: IMPALA-6818 Rethink data-stream sender/receiver startup sequencing