Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9530

Allow limiting memory consumed by preaggregation

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.0.0
    • Backend
    • None

    Description

      In some cases pre-aggregations can balloon up and consume lots of memory, forcing the merge aggregation to spill. This is often OK as long as the preaggregation is reducing the input sufficiently, since it reduces the amount of data shuffled over the network.

      However in some cases it's preferable to be more conservative with memory and just cap the size of the preaggregation to prevent it ballooning too much.

      It would be useful to add a query option to directly limit the memory consumption of the preaggregations.

      Attachments

        Issue Links

          Activity

            People

              tarmstrong Tim Armstrong
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: