[IMPALA-5485] Exchg / data-stream recv-side buffers should operate under a memory reservation

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Distributed Exec
• Target Version: Product Backlog
• Epic Color: ghx-label-7
• Assignee: Unassigned
• Reporter: Henry Robinson
• Created: 12/Jun/17 17:58
• Updated: 23/Oct/19 16:18

Description

Exchange nodes assign a fixed-size buffer to their datastream receivers that's used to smooth out differences in send / consume rates between the sender and the receiver.

These buffers should be tracked by the query memtracker, and with the new min-reservation support we should allow them to be larger than the configured minimum. Increasing the buffer size decreases the amount of time that a sender can be blocked on a receiver, and so increases query-parallelism. Queries that shuffle a lot of data can see significant speedups from larger buffers.

The buffers need to be sized based on the #senders and the #rows * #avg row size. They can dynamically expand trivially - contraction is possible, but a bit harder.

Issue Links

• depends upon: IMPALA-5832 Simplify the backend's memory ownership and transfer model
• is depended upon by: IMPALA-4834 All operators should operate within a memory constraint
• relates to: IMPALA-6524 Sender and receiver side of exchange should operate within memory constraint