Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2399

Cleanup/rethink QueryMaintenance() calls in the BE.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: None
    • Component/s: Backend

      Description

      In our BE execution, we currently call QueryMaintenance() and check for cancellation very generously, e.g., at the beginning of every GetNext() or in tight loops that could run for a long time (e.g. due to selective predicates).

      Our current pattern of usage has several problems:
      1. QueryMaintenance() is expensive, but the function name is rather opaque to this fact. It traverses the mem tracker hierarchy checking for the mem limit, and also checks the query status under a spin lock. It is not clear that these checks need to be coupled and called so frequently. We should place these checks judiciously/appropriately because QueryMaintenance() can become a performance bottleneck especially in subplan execution.

      2. We always couple QueryMaintenance() with checking for cancellation. There is an existing TODO in the code to address this (see runtime-state.cc RuntimeState::CheckQueryState()).

      See these code reviews for additional discussion:
      http://gerrit.cloudera.org:8080/894
      http://gerrit.cloudera.org:8080/901

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kwho Michael Ho
                Reporter:
                alex.behm Alexander Behm
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: