[SYSTEMDS-2029] Perftest L2SVM icpt 1 sparse 800GB w/ codegen failing

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.0.0
• Epic Link: Perftest and QA for SystemML 1.0 release
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 29/Nov/17 21:27
• Updated: 30/Nov/17 21:52
• Resolved: 30/Nov/17 21:52

Description

Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 141 and 161 -- Error evaluating instruction: CP°ba+*°_mVar169·MATRIX·DOUBLE°_mVar170·MATRIX·DOUBLE°_mVar176·MATRIX·DOUBLE°16
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:294)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:218)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
	at org.apache.sysml.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:115)
	... 14 more
Caused by: org.apache.sysml.runtime.controlprogram.caching.CacheException: Reading of scratch_space//_p2037254_9.1.44.28//_t0/temp66_56 (656476731) failed.
	at org.apache.sysml.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:436)
	at org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:274)
	at org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:252)
	at org.apache.sysml.runtime.instructions.cp.AggregateBinaryCPInstruction.processInstruction(AggregateBinaryCPInstruction.java:71)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:264)
	... 17 more
Caused by: java.io.IOException: Failed parallel read of binary block input.
	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readBinaryBlockMatrixFromHDFS(ReaderBinaryBlockParallel.java:113)
	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:70)
	at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:204)
	at org.apache.sysml.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:168)
	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:434)
	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:59)
	at org.apache.sysml.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:977)
	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromRDD(MatrixObject.java:491)
	at org.apache.sysml.runtime.controlprogram.caching.MatrixObject.readBlobFromRDD(MatrixObject.java:59)
	at org.apache.sysml.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:426)
	... 21 more
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 46
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel.readBinaryBlockMatrixFromHDFS(ReaderBinaryBlockParallel.java:103)
	... 30 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 46
	at org.apache.sysml.runtime.matrix.data.SparseRowVector.append(SparseRowVector.java:197)
	at org.apache.sysml.runtime.matrix.data.SparseBlockMCSR.append(SparseBlockMCSR.java:264)
	at org.apache.sysml.runtime.matrix.data.MatrixBlock.appendToSparse(MatrixBlock.java:758)
	at org.apache.sysml.runtime.matrix.data.MatrixBlock.appendToSparse(MatrixBlock.java:721)
	at org.apache.sysml.runtime.io.ReaderBinaryBlockParallel$ReadFileTask.call(ReaderBinaryBlockParallel.java:183)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)