[DRILL-229] Build a Merging Recevier

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Assignee: Ben Becker
• Reporter: Jacques Nadeau
• Created: 11/Sep/13 17:44
• Updated: 31/Jul/14 06:32
• Resolved: 18/Nov/13 22:57

Description

Build a merging receiver operator which combines a number of incoming batches into a single outgoing batch.

Overview

Incoming batches are individually presorted before reaching this operator, so a priority queue is built with each value to determine which batch contains the next record. When the next value is removed from the priority queue, the underlying record is copied from the underlying ValueVectors to the outgoing VectorContainer.

The comparator for the priority queue is generated based on the supplied LogicalExpression (e.g. a single ValueVectorReadExpression).

Example

The following example illustrates a distributed count operation, where each remote fragment counts a subset of the data and the root fragment produces a sum of each count aggregate.

Data Flow

RecordReader
  |
  +-> Sort
       |
       +-> StreamingAggregate(COUNT)
              |
              +-> MergingPartitionExchange
                     |
                     +-> StreamingAggregate(SUM)
                            |
                            +-> UnionExchange
                                   |
                                   +-> Screen

Control Flow

Root Fragment
-------------
Screen
   |
   +->UnionExchange
         | | |
         | | +->AggSum
         | |      |
         | |      +->MergingReceiver
         | |
         | +--->AggSum
         |        |
         |        +->MergingReceiver
         |
         +----->AggSum
                  |
                  +->MergingReceiver
             ...

Remote Fragment
---------------
PartitionSender
       |
       +->AggCount
             |
             +->Sort
                 |
                 +->Reader