[SYSTEMDS-2382] Concurrent problem about LibMatrixMult

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.2
• Assignee: LI Guobao
• Reporter: LI Guobao
• Created: 11/Jun/18 22:31
• Updated: 13/Jun/18 21:51
• Resolved: 12/Jun/18 02:53

Description

When the worker number is set to 2 and each worker has 2 threads to launch its instruction, in the case of small batchsize, the program fails with a concurrent error. But when I set only 1 thread to each worker, everything works fine. By the way, the number of vcore is 4 on my machine.

Here is the stack:

Caused by: org.apache.sysml.runtime.DMLRuntimeException: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.sysml.runtime.matrix.data.LibMatrixMult.matrixMult(LibMatrixMult.java:221)
at org.apache.sysml.runtime.matrix.data.MatrixBlock.aggregateBinaryOperations(MatrixBlock.java:4796)
at org.apache.sysml.runtime.instructions.cp.AggregateBinaryCPInstruction.processInstruction(AggregateBinaryCPInstruction.java:73)
at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:252)
... 13 more
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 3
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.sysml.runtime.matrix.data.LibMatrixMult.matrixMult(LibMatrixMult.java:213)
... 16 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.sysml.runtime.matrix.data.LibMatrixMult.matrixMultSparseDenseMMShortLHS(LibMatrixMult.java:1305)
at org.apache.sysml.runtime.matrix.data.LibMatrixMult.matrixMultSparseDense(LibMatrixMult.java:1194)
at org.apache.sysml.runtime.matrix.data.LibMatrixMult.access$300(LibMatrixMult.java:57)
at org.apache.sysml.runtime.matrix.data.LibMatrixMult$MatrixMultTask.call(LibMatrixMult.java:3921)
... 5 more