Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1636

Generalize index-based partition pruning to allow constant expressions in IN predicates.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.1
    • Fix Version/s: Impala 2.2
    • Component/s: None

      Description

      Currently, the index-based partition pruning requires children of IN predicates to be literals, i.e., something like this:

      ... WHERE year IN (2009, 2010)
      

      However, we should also allow constant expressions such as

      ... WHERE year IN (year(now()) + 1, 2010)
      

      This use case is important because partitions are often based on time and it's very convenient to use our timestamp functions to compute a date - unfortunately, the fast path for partition pruning will then not kick in.

      The relevant part of the code is HdfsScanNode.java::canEvalUsingPartitionMd()

      ...
      } else if (expr instanceof InPredicate) {
        // Check for SlotRef [NOT] IN (Literal, ... Literal) case
        SlotRef slot = ((InPredicate)expr).getBoundSlot();
        if (slot == null) return false;
      
        for (int i = 1; i < expr.getChildren().size(); ++i) {
          Expr rhs = expr.getChild(i);
          if (!(rhs.isLiteral())) { <---- Should also accept/eval const exprs
            return false;
          } else {
            // Make sure the SlotRef column and the LiteralExpr have the same
            // PrimitiveType. If not, the expr needs to be evaluated in the BE.
            if (!hasIdenticalType(slot, rhs)) return false;
          }
        }
        return true;
      }
      ...
      

        Attachments

          Activity

            People

            • Assignee:
              dtsirogiannis Dimitris Tsirogiannis
              Reporter:
              alex.behm Alexander Behm
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: