Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6113

Limit batch size for Merge Receiver

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.12.0
    • Fix Version/s: None
    • Component/s: Execution - Flow
    • Labels:
      None

      Description

      Merge receiver has hard coded limit of 32K rows as batch. Since rows can be of varying width, it is difficult to predict the output batch size (in terms of memory) for this operator. Change this to derive row count based on actual memory available. We are introducing a new option called outputBatchSize to limit the batch size of each operator. Use the memory configured from that. Figure out the average row width of outgoing batch based on averages of batches coming from incoming streams. Limit the row count based on memory available to use and average row width.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ppenumarthy Padma Penumarthy
                Reporter:
                ppenumarthy Padma Penumarthy
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: