Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3352

Analytic function eval order changed for no reason in planner test

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • Impala 2.6.0
    • None
    • Frontend

    Description

      I think the plan is equivalent but I'm not sure why this failed.

      com.cloudera.impala.planner.PlannerTest.testAnalyticFns
Failing for the past 1 build (Since Failed#2237 )
Took 0.85 sec.
Error Message

Section PLAN of query:
select * from
  (select year, tinyint_col,
   last_value(int_col) over(partition by int_col, year order by id
   rows between 1 preceding and 1 following),
   last_value(tinyint_col) over(partition by id, year order by int_col
   range between unbounded preceding and unbounded following),
   sum(int_col) over(partition by year, tinyint_col),
   avg(int_col) over(partition by tinyint_col, id, year order by bigint_col)
   from functional.alltypes) v
where year = 2009 and tinyint_col + 1 = 1

Actual does not match expected result:
09:SELECT
|  predicates: tinyint_col + 1 = 1
|
08:ANALYTIC
|  functions: avg(int_col)
^^^^^^^^^^^^^^^^^^^^^^^^^^
|  partition by: tinyint_col, id, year
|  order by: bigint_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|
07:SORT
|  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
|
06:ANALYTIC
|  functions: sum(int_col)
|  partition by: year, tinyint_col
|
05:SORT
|  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
|
04:ANALYTIC
|  functions: last_value(int_col)
|  partition by: int_col, year
|  order by: id ASC
|  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
|
03:SORT
|  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
|
02:ANALYTIC
|  functions: last_value(tinyint_col)
|  partition by: id, year
|  order by: int_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
|
01:SORT
|  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
|
00:SCAN HDFS [functional.alltypes]
   partitions=12/24 files=12 size=238.68KB

Expected:
09:SELECT
|  predicates: tinyint_col + 1 = 1
|
08:ANALYTIC
|  functions: sum(int_col)
|  partition by: year, tinyint_col
|
07:SORT
|  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
|
06:ANALYTIC
|  functions: last_value(int_col)
|  partition by: int_col, year
|  order by: id ASC
|  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
|
05:SORT
|  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
|
04:ANALYTIC
|  functions: avg(int_col)
|  partition by: tinyint_col, id, year
|  order by: bigint_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|
03:SORT
|  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
|
02:ANALYTIC
|  functions: last_value(tinyint_col)
|  partition by: id, year
|  order by: int_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
|
01:SORT
|  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
|
00:SCAN HDFS [functional.alltypes]
   partitions=12/24 files=12 size=238.68KB

Verbose plan:
F00:PLAN FRAGMENT [UNPARTITIONED]
  09:SELECT
  |  predicates: tinyint_col + 1 = 1
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=15,8 row-size=42B cardinality=365
  |
  08:ANALYTIC
  |  functions: avg(int_col)
  |  partition by: tinyint_col, id, year
  |  order by: bigint_col ASC
  |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=15,8 row-size=42B cardinality=3650
  |
  07:SORT
  |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=15 row-size=34B cardinality=3650
  |
  06:ANALYTIC
  |  functions: sum(int_col)
  |  partition by: year, tinyint_col
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=13,6 row-size=34B cardinality=3650
  |
  05:SORT
  |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=13 row-size=26B cardinality=3650
  |
  04:ANALYTIC
  |  functions: last_value(int_col)
  |  partition by: int_col, year
  |  order by: id ASC
  |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=11,4 row-size=26B cardinality=3650
  |
  03:SORT
  |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=11 row-size=22B cardinality=3650
  |
  02:ANALYTIC
  |  functions: last_value(tinyint_col)
  |  partition by: id, year
  |  order by: int_col ASC
  |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=9,5 row-size=22B cardinality=3650
  |
  01:SORT
  |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=9 row-size=21B cardinality=3650
  |
  00:SCAN HDFS [functional.alltypes]
     partitions=12/24 files=12 size=238.68KB
     table stats: 7300 rows total
     column stats: all
     hosts=2 per-host-mem=unavailable
     tuple-ids=0 row-size=21B cardinality=3650

Stacktrace

java.lang.AssertionError: 
Section PLAN of query:
select * from
  (select year, tinyint_col,
   last_value(int_col) over(partition by int_col, year order by id
   rows between 1 preceding and 1 following),
   last_value(tinyint_col) over(partition by id, year order by int_col
   range between unbounded preceding and unbounded following),
   sum(int_col) over(partition by year, tinyint_col),
   avg(int_col) over(partition by tinyint_col, id, year order by bigint_col)
   from functional.alltypes) v
where year = 2009 and tinyint_col + 1 = 1

Actual does not match expected result:
09:SELECT
|  predicates: tinyint_col + 1 = 1
|
08:ANALYTIC
|  functions: avg(int_col)
^^^^^^^^^^^^^^^^^^^^^^^^^^
|  partition by: tinyint_col, id, year
|  order by: bigint_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|
07:SORT
|  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
|
06:ANALYTIC
|  functions: sum(int_col)
|  partition by: year, tinyint_col
|
05:SORT
|  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
|
04:ANALYTIC
|  functions: last_value(int_col)
|  partition by: int_col, year
|  order by: id ASC
|  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
|
03:SORT
|  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
|
02:ANALYTIC
|  functions: last_value(tinyint_col)
|  partition by: id, year
|  order by: int_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
|
01:SORT
|  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
|
00:SCAN HDFS [functional.alltypes]
   partitions=12/24 files=12 size=238.68KB

Expected:
09:SELECT
|  predicates: tinyint_col + 1 = 1
|
08:ANALYTIC
|  functions: sum(int_col)
|  partition by: year, tinyint_col
|
07:SORT
|  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
|
06:ANALYTIC
|  functions: last_value(int_col)
|  partition by: int_col, year
|  order by: id ASC
|  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
|
05:SORT
|  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
|
04:ANALYTIC
|  functions: avg(int_col)
|  partition by: tinyint_col, id, year
|  order by: bigint_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|
03:SORT
|  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
|
02:ANALYTIC
|  functions: last_value(tinyint_col)
|  partition by: id, year
|  order by: int_col ASC
|  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
|
01:SORT
|  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
|
00:SCAN HDFS [functional.alltypes]
   partitions=12/24 files=12 size=238.68KB

Verbose plan:
F00:PLAN FRAGMENT [UNPARTITIONED]
  09:SELECT
  |  predicates: tinyint_col + 1 = 1
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=15,8 row-size=42B cardinality=365
  |
  08:ANALYTIC
  |  functions: avg(int_col)
  |  partition by: tinyint_col, id, year
  |  order by: bigint_col ASC
  |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=15,8 row-size=42B cardinality=3650
  |
  07:SORT
  |  order by: tinyint_col ASC NULLS FIRST, id ASC NULLS FIRST, year ASC NULLS FIRST, bigint_col ASC
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=15 row-size=34B cardinality=3650
  |
  06:ANALYTIC
  |  functions: sum(int_col)
  |  partition by: year, tinyint_col
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=13,6 row-size=34B cardinality=3650
  |
  05:SORT
  |  order by: year ASC NULLS FIRST, tinyint_col ASC NULLS FIRST
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=13 row-size=26B cardinality=3650
  |
  04:ANALYTIC
  |  functions: last_value(int_col)
  |  partition by: int_col, year
  |  order by: id ASC
  |  window: ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=11,4 row-size=26B cardinality=3650
  |
  03:SORT
  |  order by: int_col ASC NULLS FIRST, year ASC NULLS FIRST, id ASC
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=11 row-size=22B cardinality=3650
  |
  02:ANALYTIC
  |  functions: last_value(tinyint_col)
  |  partition by: id, year
  |  order by: int_col ASC
  |  window: RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=9,5 row-size=22B cardinality=3650
  |
  01:SORT
  |  order by: id ASC NULLS FIRST, year ASC NULLS FIRST, int_col ASC
  |  hosts=2 per-host-mem=unavailable
  |  tuple-ids=9 row-size=21B cardinality=3650
  |
  00:SCAN HDFS [functional.alltypes]
     partitions=12/24 files=12 size=238.68KB
     table stats: 7300 rows total
     column stats: all
     hosts=2 per-host-mem=unavailable
     tuple-ids=0 row-size=21B cardinality=3650

	at org.junit.Assert.fail(Assert.java:88)
	at com.cloudera.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:623)
	at com.cloudera.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:628)
	at com.cloudera.impala.planner.PlannerTest.testAnalyticFns(PlannerTest.java:52)

      http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge/2237/testReport/junit/com.cloudera.impala.planner/PlannerTest/testAnalyticFns/

      Attachments

        Activity

          People

            Unassigned Unassigned
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: