Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4834

All operators should operate within a memory constraint

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Epic
    • Status: Open
    • Major
    • Resolution: Unresolved
    • Impala 2.9.0
    • None
    • Backend
    • Operator memory constraints
    • ghx-label-9

    Description

      Once the reservation management of IMPALA-3200 is enabled, we should start modifying more operators to operate within a memory constraint, so that once the query starts the operators can run to completion.

      This means that the bulk of the operator memory, particularly the memory that depends on the input size, should be allocated from the buffer pool. The operator should then be able to execute so long as it gets it's minimum reservation.

      It is ok for now if expr memory and smaller amounts of memory like rowbatches and small decompression buffers that are streamed through the plan isn't included in this.

      We also don't need to handle arbitrarily sized rows, only up to a configurable threshold (e.g. a query option)

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tarmstrong Tim Armstrong

            Dates

              Created:
              Updated:

              Slack

                Issue deployment