Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4263

Wrong results due to missing hash exchange believed to be redundant.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Frontend
    • Labels:

      Description

      Query plans that have an outer join followed by a grouping aggregation or an analytic function with a partition by clause may produce incorrect results. The reason is that Impala incorrectly optimizes away a hash exchange that is believed to be redundant (but actually is required).

      Example aggregation query and bad plan:

      explain select /* +straight_join */ t2.id, count(*)
      from functional.alltypes t1
      left outer join /* +shuffle */ functional.alltypessmall t2
        on t1.id = t2.id
      group by t2.id
      
      PLAN-ROOT SINK
      |
      06:EXCHANGE [UNPARTITIONED]
      |
      03:AGGREGATE [FINALIZE] <-- Missing hash exchange and merge step.
      |  output: count(*)
      |  group by: t2.id
      |
      02:HASH JOIN [LEFT OUTER JOIN, PARTITIONED]
      |  hash predicates: t1.id = t2.id
      |
      |--05:EXCHANGE [HASH(t2.id)]
      |  |
      |  01:SCAN HDFS [functional.alltypessmall t2]
      |     partitions=4/4 files=4 size=6.32KB
      |
      04:EXCHANGE [HASH(t1.id)]
      |
      00:SCAN HDFS [functional.alltypes t1]
         partitions=24/24 files=24 size=478.45KB
      

      Example analytic query and bad plan:

      explain select /* +straight_join */ count(*) over (partition by t1.id)
      from functional.alltypes t1
      right outer join /* +shuffle */ functional.alltypessmall t2
        on t1.id = t2.id
      
      PLAN-ROOT SINK
      |
      07:EXCHANGE [UNPARTITIONED]
      |
      04:ANALYTIC
      |  functions: count(*)
      |  partition by: t1.id
      |
      03:SORT
      |  order by: id ASC NULLS FIRST <-- Missing hash exchange before sort.
      |
      02:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]
      |  hash predicates: t1.id = t2.id
      |  runtime filters: RF000 <- t2.id
      |
      |--06:EXCHANGE [HASH(t2.id)]
      |  |
      |  01:SCAN HDFS [functional.alltypessmall t2]
      |     partitions=4/4 files=4 size=6.32KB
      |
      05:EXCHANGE [HASH(t1.id)]
      |
      00:SCAN HDFS [functional.alltypes t1]
         partitions=24/24 files=24 size=478.45KB
         runtime filters: RF000 -> t1.id
      

      The problem is that if grouping/partition exprs reference nullable tuples, then we need a hash exchange to bring the NULLs of outer-join non-matches together.

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit cd153d66dc20ebea7a9de7a2ef1c627fee45253c
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Wed Jan 4 22:19:38 2017 -0800

        IMPALA-4263: Fix wrong ommission of agg/analytic hash exchanges.

        The bug: Our detection of partition compatibility for
        grouping aggregations and analytic functions did not take into
        account the effect of outer joins within the same fragment.
        As a result, we used to incorrectly omit a required hash exchange.
        For example, a hash exchange + merge phase is required if the
        grouping expressions of an aggregation reference tuples
        that are made nullable within the same fragment. The exchange is
        needed to bring together NULLs produced by outer-join non-matches.

        The fix: Check that the grouping/partition exprs do not reference
        tuples that are made nullable within the same fragment.

        Testing: Planner tests pass locally.

        Change-Id: I121222179378e56836422a69451d840a012c9e54
        Reviewed-on: http://gerrit.cloudera.org:8080/5774
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Alex Behm <alex.behm@cloudera.com>

        Show
        alex.behm Alexander Behm added a comment - commit cd153d66dc20ebea7a9de7a2ef1c627fee45253c Author: Alex Behm <alex.behm@cloudera.com> Date: Wed Jan 4 22:19:38 2017 -0800 IMPALA-4263 : Fix wrong ommission of agg/analytic hash exchanges. The bug: Our detection of partition compatibility for grouping aggregations and analytic functions did not take into account the effect of outer joins within the same fragment. As a result, we used to incorrectly omit a required hash exchange. For example, a hash exchange + merge phase is required if the grouping expressions of an aggregation reference tuples that are made nullable within the same fragment. The exchange is needed to bring together NULLs produced by outer-join non-matches. The fix: Check that the grouping/partition exprs do not reference tuples that are made nullable within the same fragment. Testing: Planner tests pass locally. Change-Id: I121222179378e56836422a69451d840a012c9e54 Reviewed-on: http://gerrit.cloudera.org:8080/5774 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Alex Behm <alex.behm@cloudera.com>
        Hide
        jbapple Jim Apple added a comment -
        Show
        jbapple Jim Apple added a comment - Patch from Alexander Behm in review: https://gerrit.cloudera.org/#/c/5774/
        Hide
        alex.behm Alexander Behm added a comment -

        Michael Brown, can you check whether the query generator could have caught this issue? If not, we should file a separate JIRA to change it so such plans can be generated.

        Show
        alex.behm Alexander Behm added a comment - Michael Brown , can you check whether the query generator could have caught this issue? If not, we should file a separate JIRA to change it so such plans can be generated.

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development