Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
SystemDS 3.1
-
None
-
None
Description
Please use SAGA/runbug3.sh om charlie to reproduce the error
org.apache.sysds.runtime.DMLRuntimeException: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 31 – Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar2769·FRAME·FP64·false°Xtrain=_mVar2772·MATRIX·FP64°Ytrain=_mVar2774·MATRIX·FP64°Xtest=_mVar2776·MATRIX·FP64°Ytest=_mVar2778·MATRIX·FP64°metaList=_Var2785·LIST·UNKNOWN·false°hyperParameters=_mVar2786·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°hpForPruning=_mVar2787·MATRIX·FP64°changesByOp=_mVar2787·MATRIX·FP64°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:164) at org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:89) at org.apache.sysds.api.DMLScript.execute(DMLScript.java:461) at org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:304) at org.apache.sysds.api.DMLScript.main(DMLScript.java:198) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 1 and 31 – Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar2769·FRAME·FP64·false°Xtrain=_mVar2772·MATRIX·FP64°Ytrain=_mVar2774·MATRIX·FP64°Xtest=_mVar2776·MATRIX·FP64°Ytest=_mVar2778·MATRIX·FP64°metaList=_Var2785·LIST·UNKNOWN·false°hyperParameters=_mVar2786·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°hpForPruning=_mVar2787·MATRIX·FP64°changesByOp=_mVar2787·MATRIX·FP64°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:158) ... 16 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::s_executePipeline at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 19 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 52 and 119 – Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:162) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201) ... 20 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 – Error evaluating for program block at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:155) ... 21 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 79 and 104 – Error evaluating if statement body at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113) at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143) ... 22 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 80 and 80 – Error evaluating instruction: CP°eval°outlierBySd·SCALAR·STRING·true°hp·LIST·UNKNOWN·false°_Var3196·LIST·UNKNOWN at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106) ... 23 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::m_outlierBySd at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208) at org.apache.sysds.runtime.instructions.cp.EvalNaryCPInstruction.processInstruction(EvalNaryCPInstruction.java:173) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 26 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 39 and 87 – Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:162) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201) ... 28 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in while program block generated from while statement block between lines 52 and 74 – Error evaluating while program block at org.apache.sysds.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:139) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:155) ... 29 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 62 and 71 – Error evaluating if statement body at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113) at org.apache.sysds.runtime.controlprogram.WhileProgramBlock.execute(WhileProgramBlock.java:117) ... 30 more Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 67 and 68 – Error evaluating instruction: SPARK°uak+°_mVar3514·MATRIX·FP64°_mVar3515·MATRIX·FP64°SINGLE_BLOCK at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:291) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106) ... 31 more Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 186 in stage 146.0 failed 4 times, most recent failure: Lost task 186.3 in stage 146.0 (TID 49964) (whiskey.dm.isds.tugraz.at executor 5): java.lang.RuntimeException: Invalid block dimensions: -1360007040 14 at org.apache.sysds.runtime.matrix.data.MatrixBlock.reset(MatrixBlock.java:285) at org.apache.sysds.runtime.matrix.data.MatrixBlock.reset(MatrixBlock.java:249) at org.apache.sysds.runtime.matrix.data.LibMatrixReorg.rmempty(LibMatrixReorg.java:700) at org.apache.sysds.runtime.instructions.spark.ParameterizedBuiltinSPInstruction$RDDRemoveEmptyFunctionInMem.call(ParameterizedBuiltinSPInstruction.java:687) at org.apache.sysds.runtime.instructions.spark.ParameterizedBuiltinSPInstruction$RDDRemoveEmptyFunctionInMem.call(ParameterizedBuiltinSPInstruction.java:659) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMapToPair$1(JavaRDDLike.scala:143) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:197) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2454) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2403) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2402) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2402) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1160) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1160) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1160) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2642) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2584) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2573) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:938) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2214) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2309) at org.apache.spark.rdd.RDD.$anonfun$fold$1(RDD.scala:1183) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) at org.apache.spark.rdd.RDD.fold(RDD.scala:1177) at org.apache.spark.api.java.JavaRDDLike.fold(JavaRDDLike.scala:415) at org.apache.spark.api.java.JavaRDDLike.fold$(JavaRDDLike.scala:414) at org.apache.spark.api.java.AbstractJavaRDDLike.fold(JavaRDDLike.scala:45) at org.apache.sysds.runtime.instructions.spark.utils.RDDAggregateUtils.aggStable(RDDAggregateUtils.java:148) at org.apache.sysds.runtime.instructions.spark.AggregateUnarySPInstruction.processMatrixAggregate(AggregateUnarySPInstruction.java:110) at org.apache.sysds.runtime.instructions.spark.AggregateUnarySPInstruction.processInstruction(AggregateUnarySPInstruction.java:80) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:255) ... 34 more Caused by: java.lang.RuntimeException: Invalid block dimensions: -1360007040 14 at org.apache.sysds.runtime.matrix.data.MatrixBlock.reset(MatrixBlock.java:285) at org.apache.sysds.runtime.matrix.data.MatrixBlock.reset(MatrixBlock.java:249) at org.apache.sysds.runtime.matrix.data.LibMatrixReorg.rmempty(LibMatrixReorg.java:700) at org.apache.sysds.runtime.instructions.spark.ParameterizedBuiltinSPInstruction$RDDRemoveEmptyFunctionInMem.call(ParameterizedBuiltinSPInstruction.java:687) at org.apache.sysds.runtime.instructions.spark.ParameterizedBuiltinSPInstruction$RDDRemoveEmptyFunctionInMem.call(ParameterizedBuiltinSPInstruction.java:659) at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMapToPair$1(JavaRDDLike.scala:143) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:197) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)