[HIVE-9225] Windowing functions are not executing efficiently when the window is identical

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.14.1
• Component/s: PTF-Windowing

Environment

Linux

• Release Note: Streaming mode improves performance in such use cases.
• Assignee: Unassigned
• Reporter: Illya Yalovyy
• Created: 30/Dec/14 03:05
• Updated: 10/Jun/15 21:58
• Resolved: 10/Jun/15 21:58

Description

Hive optimizer and the runtime are not smart enough to recognize if the windowing is the same. Even when the window is identical, the windowing is re-executed again and cause the runtime increase proportionally to # of windows.

Example:

select code,min(emp) over (partition by code order by emp  range between current row and 300000000 following)from sample_big limit 10;

Time taken: 1h:36m:12s

select code,
min(emp) over (partition by code order by emp  range between current row and 300000000 following),
max(emp) over (partition by code order by emp  range between current row and 300000000 following),
min(salary) over (partition by code order by emp  range between current row and 300000000 following),
max(salary) over (partition by code order by emp  range between current row and 300000000 following)
from sample_big limit 10;

Time taken: 4h:0m:37s