Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7301 Resource management for exchange node and datastream sender
  3. IMPALA-5485

Exchg / data-stream recv-side buffers should operate under a memory reservation

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Distributed Exec
    • Labels:
      None

      Description

      Exchange nodes assign a fixed-size buffer to their datastream receivers that's used to smooth out differences in send / consume rates between the sender and the receiver.

      These buffers should be tracked by the query memtracker, and with the new min-reservation support we should allow them to be larger than the configured minimum. Increasing the buffer size decreases the amount of time that a sender can be blocked on a receiver, and so increases query-parallelism. Queries that shuffle a lot of data can see significant speedups from larger buffers.

      The buffers need to be sized based on the #senders and the #rows * #avg row size. They can dynamically expand trivially - contraction is possible, but a bit harder.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                henryr Henry Robinson
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: