[TAJO-2000] BSTIndex can cause OOM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.12.0, 0.11.1
• Component/s: Data Shuffle, Physical Operator
• Assignee: Jinho Kim
• Reporter: Jinho Kim
• Created: 02/Dec/15 07:34
• Updated: 07/Dec/15 06:39
• Resolved: 07/Dec/15 05:10

Description

BSTIndex writer collect the key and offsets for range shuffle. if keys is unique, collector needs large memory.
In case of sorted dataset, dataset don’t need to keep in the memory. It should be written in the file immediately

Here is the stack trace

Thread 63932: (state = BLOCKED)
- org.apache.tajo.storage.VTuple.isBlankOrNull(int) @bci=15, line=67 (Compiled frame; information may be imprecise)
- org.apache.tajo.storage.BaseTupleComparator.compare(org.apache.tajo.storage.Tuple, org.apache.tajo.storage.Tuple) @bci=32, line=112 (Compiled frame)
- org.apache.tajo.storage.BaseTupleComparator.compare(java.lang.Object, java.lang.Object) @bci=9, line=37 (Compiled frame)
- java.util.TreeMap.getEntryUsingComparator(java.lang.Object) @bci=29, line=376 (Compiled frame)
- java.util.TreeMap.getEntry(java.lang.Object) @bci=9, line=345 (Compiled frame)
- java.util.TreeMap.containsKey(java.lang.Object) @bci=2, line=232 (Compiled frame)
- org.apache.tajo.storage.index.bst.BSTIndex$BSTIndexWriter$KeyOffsetCollector.put(org.apache.tajo.storage.Tuple, long) @bci=5, line=263 (Compiled frame)
- org.apache.tajo.storage.index.bst.BSTIndex$BSTIndexWriter.write(org.apache.tajo.storage.Tuple, long) @bci=88, line=143 (Compiled frame)
- org.apache.tajo.engine.planner.physical.RangeShuffleFileWriteExec.next() @bci=78, line=108 (Compiled frame)
- org.apache.tajo.worker.TaskImpl.run() @bci=99, line=402 (Interpreted frame)
- org.apache.tajo.worker.TaskContainer.run() @bci=149, line=65 (Interpreted frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Interpreted frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)