Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Issues found:
+ Key container allocation issue in HashTable
Currently we allocate 2^16 records capacity memory for "hashValues" and "links" vectors in each BatchHolder, but for "key" holders we use the allocateNew which by default allocates low capacity memory compared to "hashValues" and "links" vector capacities (incase of Integer key, capacity is 4096 records). This causes "key" holders to fill up much sooner even though "hashValues" and "links" vectors still have lot of free entries. As a result we create more BatchHolders than required causing wasted space in "links" and "hashValues" vectors in each BatchHolder. And for each new BatchHolder we create a SV4 vector in HashJoinHelper which is another overhead.
+ Allocation issues in HashAggTemplate
HashAggTemplate has its own BatchHolders which has vectors allocated using allocateNew (i.e small capacity). Whenever a BatchHolder in HashAggTemplate reaches its capacity, we add a new BatchHolder in HashTable. This causes the HashTable BatchHolders to be not space efficient.
+ Update the HashAggTemplate.outputCurrentBatch to consider cases where all entries in a single BatchHolder are can't be copied over to output vectors in a single pass (output vectors capacity is lower than the number of records in BatchHolder)
+ Lazy BatchHolder creation for both HashAgg and HashTable
Don't allocate the BatchHolder until first put request is received. This way we don't waste space in fragments which don't receive any input records. This is possible when the group by key has very few distinct values (such as shipmode in TPCH) only few fragments receive the data. Our current parallelization code is not considering the distinct values when parallelizing hash exchanges.