[DRILL-5715] Performance of refactored HashAgg operator regressed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.12.0
• Component/s: Execution - Codegen
• Labels: performance regression

Environment

10-node RHEL 6.4 (32 Core, 256GB RAM)

• Assignee: Boaz Ben-Zvi
• Reporter: Kunal Khatua
• Reviewer: Paul Rogers
• Created: 10/Aug/17 21:10
• Updated: 20/Sep/17 23:12
• Resolved: 20/Sep/17 23:12

Description

When running the following simple HashAgg-based query on a TPCH-table - Lineitem with 6Billion rows on a 10 node setup (with a single partition to disable any possible spilling to disk)

select count(*) 
from (
  select l_quantity
    , count(l_orderkey) 
  from lineitem 
  group by l_quantity 
)  

the runtime increased from 7.378 sec to 11.323 sec [reported by the JDBC client].

To disable spill-to-disk in Drill-1.11.0, the drill-override.conf was modified to

drill.exec.hashagg.num_partitions : 1

Attached are two profiles
Drill 1.10.0 : 2675cc73-9481-16e0-7d21-5f1338611e5f.sys.drill
Drill 1.11.0 : 2675de42-3789-47b8-29e8-c5077af136db.sys.drill

A separate run was done for both scenarios with the planner.width.max_per_node=10 and profiled with YourKit.

Image snippets are attached, indicating the hotspots in both builds:
Drill 1.10.0 :
Profile: 26736242-d084-6604-aac9-927e729da755.sys.drill
CallTree: drill-1.10.0_callTree.png
HotSpot: drill-1.10.0_hotspot.png

Drill 1.11.0 :
Profile: 26736615-9e86-dac9-ad77-b022fd791f67.sys.drill
CallTree: drill-1.11.0_callTree.png
HotSpot: drill-1.11.0_hotspot.png

Issue Links

• is part of: DRILL-5694 hash agg spill to disk, second phase OOM