[HIVE-20656] Sensible defaults: Map aggregation memory configs are too aggressive

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 28/Sep/18 22:54
• Updated: 17/Nov/22 08:55
• Resolved: 09/Mar/19 02:53

Description

The defaults for the following configs seems to be too aggressive. In java this can easily lead to several full GC pauses whose memory cannot be reclaimed.

HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
    "Portion of total memory to be used by map-side group aggregation hash table"),
HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold", (float) 0.9,
    "The max memory to be used by map-side group aggregation hash table.\n" +
    "If the memory usage is higher than this number, force to flush data"),

 

We can be little bit conservative for these configs to avoid getting into GC pause. 

Issue Links

• supercedes: HIVE-20364 Update default for hive.map.aggr.hash.min.reduction