Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
SystemDS 3.1
-
None
-
None
Description
The task-parallel experiments in the spark cluster are failing with the following exception. To reproduce the error please clone the directory and use the run.sh with a fresh SystemDS jar
https://github.com/damslab/reproducibility/tree/master/tmp-SAGA/TaskParallelExperiments
An Error Occurred : DMLRuntimeException -- org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 68 and 71 -- Error evaluating instruction: CP°fcall°.builtinNS°s_topk_cleaning°true°21°1°dataTrain=trainData·FRAME·FP64·false°dataTest=testData·FRAME·STRING·false°metaData=metaInfo·FRAME·STRING·false°primitives=primitives·FRAME·FP64·false°parameters=param·FRAME·FP64·false°evaluationFunc=evalClassification·SCALAR·STRING·true°evalFunHp=_mVar8018·MATRIX·FP64°topK=3·SCALAR·INT64·true°expectedIncrease=10·SCALAR·INT64·true°max_iter=15·SCALAR·INT64·true°lq=0.1·SCALAR·FP64·true°uq=0.9·SCALAR·FP64·true°resource_val=50·SCALAR·INT64·true°cv=true·SCALAR·BOOLEAN·true°cvk=3·SCALAR·INT64·true°sample=1·SCALAR·INT64·true°isLastLabel=true·SCALAR·BOOLEAN·true°enablePruning=false·SCALAR·BOOLEAN·true°seed=42·SCALAR·INT64·true°correctTypos=false·SCALAR·BOOLEAN·true°refSol=_fVar8019·FRAME·STRING·false°topKPipelines DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 68 and 71 -- Error evaluating instruction: CP°fcall°.builtinNS°s_topk_cleaning°true°21°1°dataTrain=trainData·FRAME·FP64·false°dataTest=testData·FRAME·STRING·false°metaData=metaInfo·FRAME·STRING·false°primitives=primitives·FRAME·FP64·false°parameters=param·FRAME·FP64·false°evaluationFunc=evalClassification·SCALAR·STRING·true°evalFunHp=_mVar8018·MATRIX·FP64°topK=3·SCALAR·INT64·true°expectedIncrease=10·SCALAR·INT64·true°max_iter=15·SCALAR·INT64·true°lq=0.1·SCALAR·FP64·true°uq=0.9·SCALAR·FP64·true°resource_val=50·SCALAR·INT64·true°cv=true·SCALAR·BOOLEAN·true°cvk=3·SCALAR·INT64·true°sample=1·SCALAR·INT64·true°isLastLabel=true·SCALAR·BOOLEAN·true°enablePruning=false·SCALAR·BOOLEAN·true°seed=42·SCALAR·INT64·true°correctTypos=false·SCALAR·BOOLEAN·true°refSol=_fVar8019·FRAME·STRING·false°topKPipelines DMLRuntimeException -- error executing function .builtinNS::s_topk_cleaning DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 29 and 127 -- Error evaluating function program block DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 112 and 118 -- Error evaluating instruction: CP°fcall°./scripts/pipelines/scripts/enumerateLogical.dml°enumerateLogical°true°18°5°X=eXtrain·MATRIX·FP64°y=eYtrain·MATRIX·FP64°Xtest=eXtest·MATRIX·FP64°ytest=eYtest·MATRIX·FP64°initial_population=logical·FRAME·UNKNOWN·false°refSol=refSol·FRAME·STRING·false°seed=42·SCALAR·INT64·true°max_iter=15·SCALAR·INT64·true°metaList=_Var2481·LIST·FP64·false°evaluationFunc=evalClassification·SCALAR·STRING·true°evalFunHp=evalFunHp·MATRIX·FP64°primitives=primitives·FRAME·UNKNOWN·false°param=parameters·FRAME·UNKNOWN·false°dirtyScore=_Var2482·SCALAR·FP64·false°cv=true·SCALAR·BOOLEAN·true°cvk=3·SCALAR·INT64·true°verbose=false·SCALAR·BOOLEAN·true°ctx=ctx·LIST·UNKNOWN·false°bestLogical°bestHp°con°refChanges°acc DMLRuntimeException -- error executing function ./scripts/pipelines/scripts/enumerateLogical.dml::enumerateLogical DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 55 and 221 -- Error evaluating function program block DMLRuntimeException -- ERROR: Runtime error in while program block generated from while statement block between lines 115 and 202 -- Error evaluating while program block DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 118 and 124 -- Error evaluating instruction: CP°fcall°./scripts/builtin/bandit.dml°run_with_hyperparam°true°16°4°ph_pip=_fVar80660·FRAME·FP64·false°X=X·MATRIX·FP64°Y=y·MATRIX·FP64°Xtest=Xtest·MATRIX·FP64°Ytest=ytest·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°evaluationFunc=evalClassification·SCALAR·STRING·true°evalFunHp=evalFunHp·MATRIX·FP64°param=param·FRAME·UNKNOWN·false°cv=true·SCALAR·BOOLEAN·true°cvk=3·SCALAR·INT64·true°seed=42·SCALAR·INT64·true°default=true·SCALAR·BOOLEAN·true°r_i=1·SCALAR·INT64·true°ref=0·SCALAR·INT64·true°enablePruning=false·SCALAR·BOOLEAN·true°outPip°outHp°p°refChanges DMLRuntimeException -- error executing function ./scripts/builtin/bandit.dml::run_with_hyperparam DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 215 and 342 -- Error evaluating function program block DMLRuntimeException -- PARFOR: Failed to execute loop in parallel. SparkException -- Job aborted due to stage failure: Task 8 in stage 0.0 failed 4 times, most recent failure: Lost task 8.3 in stage 0.0 (TID 49) (lima.dm.isds.tugraz.at executor 2): org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113) at org.apache.sysds.runtime.controlprogram.parfor.ParWorker.executeSetTask(ParWorker.java:154) at org.apache.sysds.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:121) at org.apache.sysds.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:98) at org.apache.sysds.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:48) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMapToPair$1(JavaRDDLike.scala:143) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364) at scala.collection.AbstractIterator.to(Iterator.scala:1431) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at scala.collection.AbstractIterator.toArray(Iterator.scala:1431) at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1030) at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2254) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°./scripts/builtin/bandit.dml°crossV°true°10°5°X=X·MATRIX·FP64°y=Y·MATRIX·FP64°cvk=3·SCALAR·INT64·true°evalFunHp=evalFunHp·MATRIX·FP64°pipList=_Var108379·LIST·UNKNOWN·false°metaList=metaList2·LIST·UNKNOWN·false°hpForPruning=hpForPruning·MATRIX·FP64°changesByOp=changesByOp·MATRIX·FP64°evalFunc=evalClassification·SCALAR·STRING·true°ref=0·SCALAR·INT64·true°accuracy°evalHp°hpForPruning°changesByOp°changesByPip at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106) ... 33 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function ./scripts/builtin/bandit.dml::crossV at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 36 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:162) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201) ... 37 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in while program block generated from while statement block between lines 0 and 0 -- Error evaluating while program block at org.apache.sysds.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:139) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:155) ... 38 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113) at org.apache.sysds.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:117) ... 39 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar112337·FRAME·UNKNOWN·false°Xtrain=trainX·MATRIX·FP64°Ytrain=trainy·MATRIX·FP64°Xtest=testX·MATRIX·FP64°Ytest=testy·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°hyperParameters=_mVar112339·MATRIX·FP64°hpForPruning=hpForPruning·MATRIX·FP64°changesByOp=changesByOp·MATRIX·FP64°flagsCount=_Var112341·SCALAR·UNKNOWN·false°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106) ... 40 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::s_executePipeline at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 43 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:162) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201) ... 44 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:155) ... 45 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°matrixToList°true°7°4°X=Xtrain·MATRIX·FP64°Y=Ytrain·MATRIX·FP64°mask=mask·MATRIX·FP64°FD=_mVar108510·MATRIX·FP64°p=_mVar108512·MATRIX·FP64°flagsCount=flagsCount·SCALAR·INT64·false°op=op·SCALAR·STRING·false°hp°dataFlag°yFlag°executeFlag at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143) ... 46 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::matrixToList at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 49 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:162) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201) ... 50 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°applyDataFlag°true°3°2°X=X·MATRIX·FP64°mask=mask·MATRIX·FP64°dataFlag=_Var108516·SCALAR·INT64·false°X°executeFlag at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:155) ... 51 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::applyDataFlag at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 54 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:162) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201) ... 55 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:155) ... 56 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Failed to evaluate the IF predicate. at org.apache.sysds.runtime.controlprogram.IfProgramBlock.executePredicate(IfProgramBlock.java:147) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:97) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106) ... 57 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating instruction: CP°uak+°mask·MATRIX·FP64°_Var99435·SCALAR·FP64°1 at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executePredicateInstructions(ProgramBlock.java:212) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executePredicate(ProgramBlock.java:182) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.executePredicate(IfProgramBlock.java:144) ... 59 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: Reading of scratch_space///_p3093380_129.27.206.4//_t0/temp3286_8 (1189334860) failed. at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireReadIntern(CacheableData.java:600) at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:513) at org.apache.sysds.runtime.controlprogram.context.ExecutionContext.getMatrixInput(ExecutionContext.java:337) at org.apache.sysds.runtime.instructions.cp.AggregateUnaryCPInstruction.processInstruction(AggregateUnaryCPInstruction.java:221) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 62 more Caused by: java.io.IOException: File scratch_space/_p3093380_129.27.206.4/_t0/temp3286_8 does not exist on HDFS/LFS. at org.apache.sysds.runtime.io.MatrixReader.checkValidInputFile(MatrixReader.java:119) at org.apache.sysds.runtime.io.ReaderBinaryBlockParallel.readMatrixFromHDFS(ReaderBinaryBlockParallel.java:71) at org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:236) at org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:189) at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:445) at org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:68) at org.apache.sysds.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:1084) at org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireReadIntern(CacheableData.java:582) ... 66 moreDriver stacktrace: DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°./scripts/builtin/bandit.dml°crossV°true°10°5°X=X·MATRIX·FP64°y=Y·MATRIX·FP64°cvk=3·SCALAR·INT64·true°evalFunHp=evalFunHp·MATRIX·FP64°pipList=_Var108379·LIST·UNKNOWN·false°metaList=metaList2·LIST·UNKNOWN·false°hpForPruning=hpForPruning·MATRIX·FP64°changesByOp=changesByOp·MATRIX·FP64°evalFunc=evalClassification·SCALAR·STRING·true°ref=0·SCALAR·INT64·true°accuracy°evalHp°hpForPruning°changesByOp°changesByPip DMLRuntimeException -- error executing function ./scripts/builtin/bandit.dml::crossV DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block DMLRuntimeException -- ERROR: Runtime error in while program block generated from while statement block between lines 0 and 0 -- Error evaluating while program block DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar112337·FRAME·UNKNOWN·false°Xtrain=trainX·MATRIX·FP64°Ytrain=trainy·MATRIX·FP64°Xtest=testX·MATRIX·FP64°Ytest=testy·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°hyperParameters=_mVar112339·MATRIX·FP64°hpForPruning=hpForPruning·MATRIX·FP64°changesByOp=changesByOp·MATRIX·FP64°flagsCount=_Var112341·SCALAR·UNKNOWN·false°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip DMLRuntimeException -- error executing function .builtinNS::s_executePipeline DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°matrixToList°true°7°4°X=Xtrain·MATRIX·FP64°Y=Ytrain·MATRIX·FP64°mask=mask·MATRIX·FP64°FD=_mVar108510·MATRIX·FP64°p=_mVar108512·MATRIX·FP64°flagsCount=flagsCount·SCALAR·INT64·false°op=op·SCALAR·STRING·false°hp°dataFlag°yFlag°executeFlag DMLRuntimeException -- error executing function .builtinNS::matrixToList DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°applyDataFlag°true°3°2°X=X·MATRIX·FP64°mask=mask·MATRIX·FP64°dataFlag=_Var108516·SCALAR·INT64·false°X°executeFlag DMLRuntimeException -- error executing function .builtinNS::applyDataFlag DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Failed to evaluate the IF predicate. DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating instruction: CP°uak+°mask·MATRIX·FP64°_Var99435·SCALAR·FP64°1 DMLRuntimeException -- Reading of scratch_space///_p3093380_129.27.206.4//_t0/temp3286_8 (1189334860) failed. IOException -- File scratch_space/_p3093380_129.27.206.4/_t0/temp3286_8 does not exist on HDFS/LFS.