Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5839

Handle Empty Batches in Merge Receiver

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.11.0
    • Fix Version/s: 1.12.0
    • Component/s: Execution - Flow
    • Labels:
      None

      Description

      merge receiver throws an exception when it receives first batch as empty batch (no rows and no schema) from any of the senders. Problem is that the operator expects at least one batch with schema (0 rows is ok, 0 columns is not) from each of its senders.

      The way algorithm works is as follows:
      Get the first batch from each of the senders.
      Create hyper vector container with this first batch from each of the senders.
      Add the batches from senders to the priority queue
      Pop from priority queue, get the index for the current batch from that sender,
      and use that to copy from the hyper vector to the outgoing vector
      When the end of batch from a sender is reached, load the next batch from the sender.
      Stop when there are no more batches from any of the senders.

      If any of the senders do not send first batch with schema and if we skip adding that batch to the hyper vector, hyper vector is not setup correctly and all the offsets from selection vector to individual batches from senders with in the hyper vector are messed up.

      Fix for this problem is when we receive empty batch from any of the senders, create dummy batch with schema from one of the other senders and add it to the hyper vector.

      If all senders send empty first batches, we just return NONE to downstream operator.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ppenumarthy Padma Penumarthy
                Reporter:
                ppenumarthy Padma Penumarthy
                Reviewer:
                Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: