[SYSTEMDS-3417] IndexOutOfBounds due to int overflow on replace

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: SystemDS 3.1
• Fix Version/s: SystemDS 3.1
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 04/Aug/22 21:33
• Updated: 15/Mar/23 19:14
• Resolved: 05/Aug/22 13:33

Description

An Error Occurred : 
      DMLRuntimeException -- org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 2 and 32 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar2850·FRAME·FP64·false°Xtrain=_mVar2852·MATRIX·FP64°Ytrain=_mVar2853·MATRIX·FP64°Xtest=_mVar2855·MATRIX·FP64°Ytest=_mVar2856·MATRIX·FP64°metaList=_Var2863·LIST·UNKNOWN·false°hyperParameters=_mVar2865·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°hpForPruning=_mVar2866·MATRIX·FP64°changesByOp=_mVar2866·MATRIX·FP64°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 2 and 32 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar2850·FRAME·FP64·false°Xtrain=_mVar2852·MATRIX·FP64°Ytrain=_mVar2853·MATRIX·FP64°Xtest=_mVar2855·MATRIX·FP64°Ytest=_mVar2856·MATRIX·FP64°metaList=_Var2863·LIST·UNKNOWN·false°hyperParameters=_mVar2865·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°hpForPruning=_mVar2866·MATRIX·FP64°changesByOp=_mVar2866·MATRIX·FP64°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip
      DMLRuntimeException -- error executing function .builtinNS::s_executePipeline
      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 52 and 119 -- Error evaluating function program block
      DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
      DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 79 and 104 -- Error evaluating if statement body 
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 80 and 80 -- Error evaluating instruction: CP°eval°imputeByMean·SCALAR·STRING·true°hp·LIST·UNKNOWN·false°_Var3008·LIST·UNKNOWN
      DMLRuntimeException -- error executing function .builtinNS::m_imputeByMean
      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 36 and 63 -- Error evaluating function program block
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 62 and 62 -- Error evaluating instruction: CP°fcall°.builtinNS°m_imputeByMeanApply°false°2°1°X=X·MATRIX·FP64°imputedVec=imputedVec·MATRIX·FP64°X
      DMLRuntimeException -- error executing function .builtinNS::m_imputeByMeanApply
      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 36 and 43 -- Error evaluating function program block
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 40 and 42 -- Error evaluating instruction: CP°replace°pattern=NaN°replacement=0°target=X°_mVar3096·MATRIX·FP64
ArrayIndexOutOfBoundsException -- Index -2147483648 out of bounds for length 2147483632

After further debugging, this issue originates from the deserialization of large dense blocks:

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -2147483648 out of bounds for length 2147483632
	at org.apache.sysds.runtime.util.FastBufferedDataInputStream.readDoubleArray(FastBufferedDataInputStream.java:172)
	at org.apache.sysds.runtime.matrix.data.MatrixBlock.readDenseBlock(MatrixBlock.java:2033)
	at org.apache.sysds.runtime.matrix.data.MatrixBlock.readFields(MatrixBlock.java:2005)
	at org.apache.sysds.runtime.util.LocalFileUtils.readWritableFromLocal(LocalFileUtils.java:133)
	at org.apache.sysds.runtime.util.LocalFileUtils.readCacheBlockFromLocal(LocalFileUtils.java:113)
	at org.apache.sysds.runtime.controlprogram.caching.LazyWriteBuffer.readBlock(LazyWriteBuffer.java:164)
	at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromCache(MatrixObject.java:419)
	at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromCache(MatrixObject.java:1)
	at org.apache.sysds.runtime.controlprogram.caching.CacheableData.restoreBlobIntoMemory(CacheableData.java:1002)
	at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquire(CacheableData.java:1187)
	at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireReadIntern(CacheableData.java:601)
	at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:505)
	at org.apache.sysds.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:336)
	at org.apache.sysds.runtime.instructions.cp.MatrixIndexingCPInstruction.processInstruction(MatrixIndexingCPInstruction.java:69)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255)
	... 7 more