Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8483

Make coordinator fragment lighter-weight

    Details

    • Epic Color:
      ghx-label-1

      Description

      it's possible to write queries with an arbitrarily expensive coordinator fragment. One example is a query with a set of unpartitioned analytic functions each with a different ORDER BY, which results in many SORT nodes in the coordinator fragment.

      This is a problem for dedicated coordinators because it makes the resource consumption unpredictable.

      It would be useful if we could offload that work to executors and guarantee that only "lightweight' operators are part of the coordinator fragment, e.g. operators that take O(#rows returned * log(#executors)) time and O(#executors) memory - regular exchanges, merging exchanges, TOP-N, non-grouping aggregation, etc.

      I think this can be done in general by inserting an additional exchange on top of the "expensive" part of a coordinator fragment, and then ensuring that that fragment gets scheduled on an executor.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bikramjeet.vig Bikramjeet Vig
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: