Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5441

Send larger row batches over the wire

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 2.10.0
    • None
    • Perf Investigation
    • None
    • ghx-label-5

    Description

      Our on-the-wire row batch size is the same as the in-memory size (1024 rows by default). It might make sense to increase the wire-size to reduce the RPC-per-row overhead, and decrease context-switching in the receiver.

      KRPC makes it quite natural to do that: each row batch can be serialized as a sidecar in memory-size batches. The receiver can then read each batch in turn as though it were sent individually, without any need to stitch together (or split up) serialized batches.

      Attachments

        Issue Links

          depends upon

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2567 KRPC milestone 1

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: