Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4834

All operators should operate within a memory constraint

    XMLWordPrintableJSON

    Details

    • Type: Epic
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: None
    • Component/s: Backend
    • Epic Name:
      Operator memory constraints
    • Target Version:
    • Epic Color:
      ghx-label-9

      Description

      Once the reservation management of IMPALA-3200 is enabled, we should start modifying more operators to operate within a memory constraint, so that once the query starts the operators can run to completion.

      This means that the bulk of the operator memory, particularly the memory that depends on the input size, should be allocated from the buffer pool. The operator should then be able to execute so long as it gets it's minimum reservation.

      It is ok for now if expr memory and smaller amounts of memory like rowbatches and small decompression buffers that are streamed through the plan isn't included in this.

      We also don't need to handle arbitrarily sized rows, only up to a configurable threshold (e.g. a query option)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: