[HIVE-4103] Remove System.gc() call from the map-join local-task loop

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Release Note: Remove System.gc() calls from HashMapWrapper::isAbort() to avoid slow-downs during local task of the map-join
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 01/Mar/13 21:53
• Updated: 16/May/13 21:10
• Resolved: 20/Apr/13 20:52

Description

Hive's HashMapWrapper calls System.gc() twice within the HashMapWrapper::isAbort() which produces a significant slow-down during the loop.

2013-03-01 04:54:28 The gc calls took 677 ms
2013-03-01 04:54:28     Processing rows:        200000  Hashtable size: 199999  Memory usage:   62955432        rate:   0.033
2013-03-01 04:54:31 The gc calls took 956 ms
2013-03-01 04:54:31     Processing rows:        300000  Hashtable size: 299999  Memory usage:   90826656        rate:   0.048
2013-03-01 04:54:33 The gc calls took 967 ms
2013-03-01 04:54:33     Processing rows:        384160  Hashtable size: 384160  Memory usage:   114412712       rate:   0.06