[IMPALA-1978] Improve DataStreamSender::Send() in case of broadcasts

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: Product Backlog
• Fix Version/s: Product Backlog
• Labels: performance ramp-up
• Assignee: Unassigned
• Reporter: Ippokratis Pandis
• Created: 06/May/15 01:06
• Updated: 04/Jan/17 23:58
• Resolved: 15/May/15 23:41

Description

When a DataStreamSender broadcasts batches it iterates over all the channels in the same order with all other senders. But SendBatch() will block, potentially blocking all N-1 senders, see code below.

In DataStreamSender::Send():

    // SendBatch() will block if there are still in-flight rpcs (and those will
    // reference the previously written thrift batch)
    for (int i = 0; i < channels_.size(); ++i) {
      RETURN_IF_ERROR(channels_[i]->SendBatch(current_thrift_batch_));
    }

That may lead to underutilization of resources, is not efficient nor scalable. We can improve it by for example shuffling the order of channels.