Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5669

Partial sort node should operate within a memory constraint

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: None
    • Component/s: Backend

      Description

      A change currently in review, IMPALA-5498, adds a new exec node PartialSortNode. Initially, it will just allocate memory up to the query memory limit, but once the new buffer management work in IMPALA-3200 goes in, it should be modified to operate within a memory constraint.

      PartialSortNode can operate with essentially any amount of memory, with the tradeoff that a smaller limit leads to a "lower quality", more random sort. We should investigate the performance implications of different limits, and consider making the limit configurable.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            twmarshall Thomas Tauber-Marshall
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development