Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3352

Analytic function eval order changed for no reason in planner test

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.6.0
    • Fix Version/s: None
    • Component/s: Frontend

      Description

      I think the plan is equivalent but I'm not sure why this failed.

      com.cloudera.impala.planner.PlannerTest.testAnalyticFns
      Failing for the past 1 build (Since Failed#2237 )
      Took 0.85 sec.
      Error Message
      
      Section PLAN of query:
      select * from
        (select year, tinyint_col,
         last_value(int_col) over(partition by int_col, year order by id
         rows between 1 preceding and 1 following),
         last_value(tinyint_col) over(partition by id, year order by int_col
         range between unbounded preceding and unbounded following),
         sum(int_col) over(partition by year, tinyint_col),
         avg(int_col) over(partition by tinyint_col, id, year order by bigint_col)
         from functional.alltypes) v
      where year = 2009 and tinyint_col + 1 = 1
      
      Actual does not match expected result:
      09:SELECT
      |  predicates: tinyint_col + 1 = 1
      |
      08:ANALYTIC
      |  functions: avg(int_col)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^
      |  partition by: tinyint_col, id, year
      |  order by: bigint_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
      |
      07:SORT
      |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
      |
      06:ANALYTIC
      |  functions: sum(int_col)
      |  partition by: year, tinyint_col
      |
      05:SORT
      |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
      |
      04:ANALYTIC
      |  functions: last_value(int_col)
      |  partition by: int_col, year
      |  order by: id ASC
      |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
      |
      03:SORT
      |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
      |
      02:ANALYTIC
      |  functions: last_value(tinyint_col)
      |  partition by: id, year
      |  order by: int_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
      |
      01:SORT
      |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
      |
      00:SCAN HDFS [functional.alltypes]
         partitions=12/24 files=12 size=238.68KB
      
      Expected:
      09:SELECT
      |  predicates: tinyint_col + 1 = 1
      |
      08:ANALYTIC
      |  functions: sum(int_col)
      |  partition by: year, tinyint_col
      |
      07:SORT
      |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
      |
      06:ANALYTIC
      |  functions: last_value(int_col)
      |  partition by: int_col, year
      |  order by: id ASC
      |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
      |
      05:SORT
      |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
      |
      04:ANALYTIC
      |  functions: avg(int_col)
      |  partition by: tinyint_col, id, year
      |  order by: bigint_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
      |
      03:SORT
      |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
      |
      02:ANALYTIC
      |  functions: last_value(tinyint_col)
      |  partition by: id, year
      |  order by: int_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
      |
      01:SORT
      |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
      |
      00:SCAN HDFS [functional.alltypes]
         partitions=12/24 files=12 size=238.68KB
      
      Verbose plan:
      F00:PLAN FRAGMENT [UNPARTITIONED]
        09:SELECT
        |  predicates: tinyint_col + 1 = 1
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=15,8 row-size=42B cardinality=365
        |
        08:ANALYTIC
        |  functions: avg(int_col)
        |  partition by: tinyint_col, id, year
        |  order by: bigint_col ASC
        |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=15,8 row-size=42B cardinality=3650
        |
        07:SORT
        |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=15 row-size=34B cardinality=3650
        |
        06:ANALYTIC
        |  functions: sum(int_col)
        |  partition by: year, tinyint_col
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=13,6 row-size=34B cardinality=3650
        |
        05:SORT
        |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=13 row-size=26B cardinality=3650
        |
        04:ANALYTIC
        |  functions: last_value(int_col)
        |  partition by: int_col, year
        |  order by: id ASC
        |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=11,4 row-size=26B cardinality=3650
        |
        03:SORT
        |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=11 row-size=22B cardinality=3650
        |
        02:ANALYTIC
        |  functions: last_value(tinyint_col)
        |  partition by: id, year
        |  order by: int_col ASC
        |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=9,5 row-size=22B cardinality=3650
        |
        01:SORT
        |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=9 row-size=21B cardinality=3650
        |
        00:SCAN HDFS [functional.alltypes]
           partitions=12/24 files=12 size=238.68KB
           table stats: 7300 rows total
           column stats: all
           hosts=2 per-host-mem=unavailable
           tuple-ids=0 row-size=21B cardinality=3650
      
      Stacktrace
      
      java.lang.AssertionError: 
      Section PLAN of query:
      select * from
        (select year, tinyint_col,
         last_value(int_col) over(partition by int_col, year order by id
         rows between 1 preceding and 1 following),
         last_value(tinyint_col) over(partition by id, year order by int_col
         range between unbounded preceding and unbounded following),
         sum(int_col) over(partition by year, tinyint_col),
         avg(int_col) over(partition by tinyint_col, id, year order by bigint_col)
         from functional.alltypes) v
      where year = 2009 and tinyint_col + 1 = 1
      
      Actual does not match expected result:
      09:SELECT
      |  predicates: tinyint_col + 1 = 1
      |
      08:ANALYTIC
      |  functions: avg(int_col)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^
      |  partition by: tinyint_col, id, year
      |  order by: bigint_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
      |
      07:SORT
      |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
      |
      06:ANALYTIC
      |  functions: sum(int_col)
      |  partition by: year, tinyint_col
      |
      05:SORT
      |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
      |
      04:ANALYTIC
      |  functions: last_value(int_col)
      |  partition by: int_col, year
      |  order by: id ASC
      |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
      |
      03:SORT
      |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
      |
      02:ANALYTIC
      |  functions: last_value(tinyint_col)
      |  partition by: id, year
      |  order by: int_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
      |
      01:SORT
      |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
      |
      00:SCAN HDFS [functional.alltypes]
         partitions=12/24 files=12 size=238.68KB
      
      Expected:
      09:SELECT
      |  predicates: tinyint_col + 1 = 1
      |
      08:ANALYTIC
      |  functions: sum(int_col)
      |  partition by: year, tinyint_col
      |
      07:SORT
      |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
      |
      06:ANALYTIC
      |  functions: last_value(int_col)
      |  partition by: int_col, year
      |  order by: id ASC
      |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
      |
      05:SORT
      |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
      |
      04:ANALYTIC
      |  functions: avg(int_col)
      |  partition by: tinyint_col, id, year
      |  order by: bigint_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
      |
      03:SORT
      |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
      |
      02:ANALYTIC
      |  functions: last_value(tinyint_col)
      |  partition by: id, year
      |  order by: int_col ASC
      |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
      |
      01:SORT
      |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
      |
      00:SCAN HDFS [functional.alltypes]
         partitions=12/24 files=12 size=238.68KB
      
      Verbose plan:
      F00:PLAN FRAGMENT [UNPARTITIONED]
        09:SELECT
        |  predicates: tinyint_col + 1 = 1
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=15,8 row-size=42B cardinality=365
        |
        08:ANALYTIC
        |  functions: avg(int_col)
        |  partition by: tinyint_col, id, year
        |  order by: bigint_col ASC
        |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=15,8 row-size=42B cardinality=3650
        |
        07:SORT
        |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=15 row-size=34B cardinality=3650
        |
        06:ANALYTIC
        |  functions: sum(int_col)
        |  partition by: year, tinyint_col
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=13,6 row-size=34B cardinality=3650
        |
        05:SORT
        |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=13 row-size=26B cardinality=3650
        |
        04:ANALYTIC
        |  functions: last_value(int_col)
        |  partition by: int_col, year
        |  order by: id ASC
        |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=11,4 row-size=26B cardinality=3650
        |
        03:SORT
        |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=11 row-size=22B cardinality=3650
        |
        02:ANALYTIC
        |  functions: last_value(tinyint_col)
        |  partition by: id, year
        |  order by: int_col ASC
        |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=9,5 row-size=22B cardinality=3650
        |
        01:SORT
        |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
        |  hosts=2 per-host-mem=unavailable
        |  tuple-ids=9 row-size=21B cardinality=3650
        |
        00:SCAN HDFS [functional.alltypes]
           partitions=12/24 files=12 size=238.68KB
           table stats: 7300 rows total
           column stats: all
           hosts=2 per-host-mem=unavailable
           tuple-ids=0 row-size=21B cardinality=3650
      
      	at org.junit.Assert.fail(Assert.java:88)
      	at com.cloudera.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:623)
      	at com.cloudera.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:628)
      	at com.cloudera.impala.planner.PlannerTest.testAnalyticFns(PlannerTest.java:52)
      
      

      http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge/2237/testReport/junit/com.cloudera.impala.planner/PlannerTest/testAnalyticFns/

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: