[IMPALA-2167] Remove the old (unpartitioned) HJ and AGG nodes

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.2
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Target Version: Product Backlog
• Assignee: Tim Armstrong
• Reporter: Ippokratis Pandis
• Created: 01/Aug/15 22:49
• Updated: 02/Aug/17 05:28
• Resolved: 02/Aug/17 05:28

Description

Currently we maintain two version of the hash-based aggregations and joins, the (old) unpartitioned ones and the partitioned and spillable ones. The main reason we had to keep the old version it was because of the additional memory PAGG and PHJ were consuming in small-ish aggregations and joins.

But maintaining this extra code is cumbersome, error-prone and tricky to test. For example, the new PHJ supports functionality (join modes) that the old one does not support, which means that some times even though we disable PHJ we still use it, see IMPALA-1751.

If we manage to make PAGG and PHJ to consume as much memory as their unpartitioned counterparts in small-ish inputs (or a few MBs more) then there is no reason we should keep the old AGG and HJ nodes around.

Issue Links

• blocks: IMPALA-1751 Setting -enable_partitioned_hash_join=false still allows spilling for some join types.
• depends upon: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
• is related to: IMPALA-2243 SEGV in ScopedTimer during old agg node Open()
• is related to: IMPALA-1953 Add query option to allow enable/disable PHJ/PAGG per query
• is related to: IMPALA-2852 Remove old hash table