Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4571

InList predicates not being pushed to Kudu scans

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Frontend
    • Labels:

      Description

      Binary predicates are being properly pushed to Kudu scans, but InList predicates are not yet being added to the KuduScanTokenBuilder, and thus are not being pushed to the Kudu scan.

        Activity

        Hide
        mjacobs Matthew Jacobs added a comment -

        commit 9f387c858354a5c7df5fd922e731f887aa7e51f7
        Author: Matthew Jacobs <mj@cloudera.com>
        Date: Thu Dec 1 18:16:33 2016 -0800

        IMPALA-4571: Push IN predicates to Kudu

        Fixes the KuduScanNode to convert InPredicates to
        KuduPredicates and push them to the Kudu scan if possible.

        An InPredicate can be pushed to the scan if expression is of
        the exact form:
        <SlotRef> IN (<LiteralExpr>, <LiteralExpr>, ...)

        That means the InPredicate has the following properties:
        1) It has a list of literal values (i.e. not a subquery);
        All values are LiteralExprs (not SlotRefs).
        2) Not negative, i.e. only 'IN' supported, not 'NOT IN'
        3) The SlotRef is not wrapped in any casts
        4) The types of all values match the type of the SlotRef
        exactly.

        A planner test was added exercising all supported types as
        well as exprs where the values would not be supported.

        TODO: perf testing
        TODO: consider a limit on the number of list values before
        keeping the predicate on the Impala scan node
        (determine from testing)

        Change-Id: I8988d4819d20d467b48e286917e347ca00f60cf0
        Reviewed-on: http://gerrit.cloudera.org:8080/5316
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        mjacobs Matthew Jacobs added a comment - commit 9f387c858354a5c7df5fd922e731f887aa7e51f7 Author: Matthew Jacobs <mj@cloudera.com> Date: Thu Dec 1 18:16:33 2016 -0800 IMPALA-4571 : Push IN predicates to Kudu Fixes the KuduScanNode to convert InPredicates to KuduPredicates and push them to the Kudu scan if possible. An InPredicate can be pushed to the scan if expression is of the exact form: <SlotRef> IN (<LiteralExpr>, <LiteralExpr>, ...) That means the InPredicate has the following properties: 1) It has a list of literal values (i.e. not a subquery); All values are LiteralExprs (not SlotRefs). 2) Not negative, i.e. only 'IN' supported, not 'NOT IN' 3) The SlotRef is not wrapped in any casts 4) The types of all values match the type of the SlotRef exactly. A planner test was added exercising all supported types as well as exprs where the values would not be supported. TODO: perf testing TODO: consider a limit on the number of list values before keeping the predicate on the Impala scan node (determine from testing) Change-Id: I8988d4819d20d467b48e286917e347ca00f60cf0 Reviewed-on: http://gerrit.cloudera.org:8080/5316 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Internal Jenkins
        Hide
        jrussell John Russell added a comment -

        I already had general info about predicate pushdown for IN( ) but I added a few details based on the docs text. I mentioned in the EXPLAIN topic about the restriction on casting. I linked from the "Kudu Performance" subtopic over to the EXPLAIN page since that has practical info about how to recognize a good or bad Kudu plan.

        Changes are in this gerrit review: https://gerrit.cloudera.org/#/c/5649/

        Show
        jrussell John Russell added a comment - I already had general info about predicate pushdown for IN( ) but I added a few details based on the docs text. I mentioned in the EXPLAIN topic about the restriction on casting. I linked from the "Kudu Performance" subtopic over to the EXPLAIN page since that has practical info about how to recognize a good or bad Kudu plan. Changes are in this gerrit review: https://gerrit.cloudera.org/#/c/5649/

          People

          • Assignee:
            mjacobs Matthew Jacobs
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development