[HIVE-12492] MapJoin: 4 million unique integers seems to be a probe plateau

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1, 1.3.0, 2.0.0
• Fix Version/s: 2.2.0
• Component/s: Query Planning
• Labels: TODOC2.2
• Target Version/s: 2.2.0
• Assignee: Jesús Camacho Rodríguez
• Reporter: Gopal Vijayaraghavan
• Created: 22/Nov/15 13:28
• Updated: 27/Feb/24 22:23
• Resolved: 23/Feb/17 18:51

Description

After 4 million keys, the map-join implementation seems to suffer from a performance degradation.

The hashtable build & probe time makes this very inefficient, even if the data is very compact (i.e 2 ints).

Falling back onto the shuffle join or bucket map-join is useful after 2^22 items.

(Note: this fixes a statsutil issue - due to the extra clone() in the column stats path)

Issue Links

• incorporates: HIVE-16137 Default value of hive config hive.auto.convert.join.hashtable.max.entries should be set to 40m instead of 4m
• is related to: HIVE-17276 Check max shuffle size when converting to dynamically partitioned hash join
• is related to: HIVE-17521 Improve defaults for few runtime configs
• links to: RB link