[SPARK-20953] Add hash map metrics to aggregate and join

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 01/Jun/17 18:40
• Updated: 23/Aug/17 18:19
• Resolved: 01/Jul/17 02:38

Description

It would be useful if we can identify hash map collision issues early on.

We should add avg hash map probe metric to aggregate operator and hash join operator and report them. If the avg probe is greater than a specific (configurable) threshold, we should log an error at runtime.
The primary classes to look at are UnsafeFixedWidthAggregationMap, HashAggregateExec, HashedRelation, HashJoin.

Issue Links

• links to: [Github] Pull Request #18258 (viirya)

Sub-Tasks

• Add hash map metrics to aggregate: Resolved
• Add hash map metrics to join: Resolved