[SYSTEMDS-2152] Spark mapmm fails for broadcast with larger than integer dims

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.1
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 17/Feb/18 01:11
• Updated: 17/Feb/18 04:07
• Resolved: 17/Feb/18 04:07

Description

        at org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext.getBroadcastForVariable(SparkExecutionContext.java:545)
        at org.apache.sysml.runtime.instructions.spark.MapmmSPInstruction.processInstruction(MapmmSPInstruction.java:128)
        at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:264)
        ... 17 more
Caused by: java.io.IOException: org.apache.sysml.runtime.DMLRuntimeException: Matrix dimensions too large for CP runtime: 514 x 5129281161
        at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:211)
        at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:171)
        at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:434)
        at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:59)
        at org.apache.sysml.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:981)
        at org.apache.sysml.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:417)
        ... 20 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Matrix dimensions too large for CP runtime: 514 x 5129281161
        at org.apache.sysml.runtime.io.MatrixReader.createOutputMatrixBlock(MatrixReader.java:80)
        at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:59)
        at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:207)
        ... 25 more