[SYSTEMDS-3414] Pipelines failing in Hybrid execution

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: SystemDS 3.1
• Fix Version/s: SystemDS 3.1
• Assignee: Matthias Boehm
• Reporter: Shafaq Siddiqi
• Created: 01/Aug/22 14:29
• Updated: 15/Mar/23 19:13
• Resolved: 04/Aug/22 21:59

Description

Execution of an individual pipeline is failing in hybrid mode with OOM. To reproduce the error please use the attached files

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at org.apache.sysds.runtime.data.DenseBlockLFP64.allocateBlock(DenseBlockLFP64.java:45)
        at org.apache.sysds.runtime.data.DenseBlockLDRB.lambda$reset$1(DenseBlockLDRB.java:77)
        at org.apache.sysds.runtime.data.DenseBlockLDRB$$Lambda$1950/0x00007f113089f168.accept(Unknown Source)
        at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
        at java.base/java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:593)
        at org.apache.sysds.runtime.data.DenseBlockLDRB.reset(DenseBlockLDRB.java:75)
        at org.apache.sysds.runtime.data.DenseBlockLFP64.<init>(DenseBlockLFP64.java:35)
        at org.apache.sysds.runtime.data.DenseBlockFactory.createDenseBlock(DenseBlockFactory.java:94)
        at org.apache.sysds.runtime.data.DenseBlockFactory.createDenseBlock(DenseBlockFactory.java:43)
        at org.apache.sysds.runtime.data.DenseBlockFactory.createDenseBlock(DenseBlockFactory.java:37)
        at org.apache.sysds.runtime.data.DenseBlockFactory.createDenseBlock(DenseBlockFactory.java:33)
        at org.apache.sysds.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:390)
        at org.apache.sysds.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:367)
        at org.apache.sysds.runtime.matrix.data.MatrixBlock.allocateDenseBlockUnsafe(MatrixBlock.java:451)
        at org.apache.sysds.runtime.io.MatrixReader.createOutputMatrixBlock(MatrixReader.java:93)
        at org.apache.sysds.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:63)
        at org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:235)
        at org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:188)
        at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:445)
        at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:68)
        at org.apache.sysds.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:1076)
        at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromRDD(MatrixObject.java:504)
        at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromRDD(MatrixObject.java:68)
        at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireReadIntern(CacheableData.java:582)
        at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:505)
        at org.apache.sysds.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:336)
        at org.apache.sysds.runtime.instructions.cp.MatrixIndexingCPInstruction.processInstruction(MatrixIndexingCPInstruction.java:69)
        at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255)
        at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204)
        at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
        at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106)
        at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143)