[SYSTEMDS-1045] OOM issue when

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Affects Version/s: SystemML 0.11
• Fix Version/s: SystemDS 2.1
• Assignee: Unassigned
• Reporter: Arvind Surve
• Created: 13/Oct/16 21:54
• Updated: 03/Jun/21 12:18
• Resolved: 03/Jun/21 12:18

Description

OutOfMemoryError occurred while running DecisionTree

Exception in thread "Thread-88" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3714)
at org.apache.sysml.runtime.matrix.data.SparseBlockCSR.resizeCopy(SparseBlockCSR.java:726)
at org.apache.sysml.runtime.matrix.data.SparseBlockCSR.resize(SparseBlockCSR.java:716)
at org.apache.sysml.runtime.matrix.data.SparseBlockCSR.setIndexRange(SparseBlockCSR.java:436)
at org.apache.sysml.runtime.matrix.data.MatrixBlock.leftIndexingOperations(MatrixBlock.java:3962)
at org.apache.sysml.runtime.matrix.data.MatrixBlock.leftIndexingOperations(MatrixBlock.java:3879)
at org.apache.sysml.runtime.instructions.cp.MatrixIndexingCPInstruction.processInstruction(MatrixIndexingCPInstruction.java:95)
at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:305)
at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:224)
at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:168)
at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeRangeTask(ParWorker.java:236)
at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:155)
at org.apache.sysml.runtime.controlprogram.parfor.LocalParWorker.run(LocalParWorker.java:125)
at java.lang.Thread.run(Thread.java:785)

I hit this issue while running job on "Spark As A Service" with 80MB (10K x1K), and on Curly with 8G (1M x 1K) data.