[HIVEMALL-73] Reduce memory usage of each_top_k

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Assignee: Makoto Yui
• Reporter: Makoto Yui
• Created: 17/Feb/17 07:36
• Updated: 20/Feb/17 07:52
• Resolved: 20/Feb/17 07:52

Description

Better to reduce memory usage in drainQueue().

2017-02-16 05:56:22,378 FATAL [Thread-4] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at hivemall.tools.EachTopKUDTF.drainQueue(EachTopKUDTF.java:182)
    at hivemall.tools.EachTopKUDTF.close(EachTopKUDTF.java:215)
    at org.apache.hadoop.hive.ql.exec.UDTFOperator.closeOp(UDTFOperator.java:143)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:577)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
    at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:318)
    at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:237)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

Issue Links

• blocks: HIVEMALL-21 [Umbrella] Release Hivemall v0.5.0
• links to: GitHub Pull Request #47