Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5441

Send larger row batches over the wire

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: None
    • Component/s: Perf Investigation
    • Labels:
      None
    • Epic Color:
      ghx-label-5

      Description

      Our on-the-wire row batch size is the same as the in-memory size (1024 rows by default). It might make sense to increase the wire-size to reduce the RPC-per-row overhead, and decrease context-switching in the receiver.

      KRPC makes it quite natural to do that: each row batch can be serialized as a sidecar in memory-size batches. The receiver can then read each batch in turn as though it were sent individually, without any need to stitch together (or split up) serialized batches.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                henryr Henry Robinson
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: