[HIVE-12310] Update memory estimation login in TopNHash

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Query Processor
• Assignee: Hari Sankar Sivarama Subramaniyan
• Reporter: Thejas Nair
• Created: 31/Oct/15 00:03
• Updated: 16/Feb/16 23:50
• Resolved: 06/Nov/15 18:40

Description

HIVE-12084 changes TopNHash to use Runtime.getRuntime().freeMemory() for finding available memory.
However, it does not give the all the memory it could use, it ignores unallocated memory. This is because the heap size of jvm grows up to max heap size (-Xmx) as per it needs. totalMemory() gives total heap space it has allocated, and freeMemory() is the free memory within that.
See http://i.stack.imgur.com/GjuwM.png and http://stackoverflow.com/questions/3571203/what-is-the-exact-meaning-of-runtime-getruntime-totalmemory-and-freememory .
So instead of using Runtime.getRuntime().freeMemory() , I think it should use maxMemory() - totalMemory() + freeMemory()

Issue Links

• is related to: HIVE-12084 Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space