[SYSTEMDS-2954] Cleaning Enumerator fails in Spark context - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: SystemDS 2.1
Component/s: None
Labels:
None

Description

The script failed with the following exception,

An Error Occured : An Error Occured :       DMLRuntimeException -- org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block      DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 147 and 149 -- Error evaluating instruction: CP°fcall°.defaultNS°m_bandit°true°12°0°X_train=eX·MATRIX·FP64°Y_train=eY·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°targetList=targetClassification·LIST·UNKNOWN·false°lp=_fVar5386404·FRAME·UNKNOWN·false°primitives=primitives·FRAME·FP64·false°param=param·FRAME·FP64·false°k=5·SCALAR·INT64·true°R=30·SCALAR·INT64·true°verbose=true·SCALAR·BOOLEAN·true°output=file:/home/ssiddiqi/output/parfor//·SCALAR·STRING·true°index=1·SCALAR·INT64·true      DMLRuntimeException -- error executing function .defaultNS::m_bandit      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 208 and 314 -- Error evaluating function program block      DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 282 and 285 -- Error evaluating instruction: CP°fcall°.defaultNS°extractBracketWinners°true°4°2°pipeline=_sbcvar12·MATRIX·FP64°hyperparam=_sbcvar13·MATRIX·FP64°k=5·SCALAR·INT64·true°conf=lookup·FRAME·FP64·false°bracket_bestPipeline°bracket_bestHyperparams      DMLRuntimeException -- error executing function .defaultNS::extractBracketWinners      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 609 and 630 -- Error evaluating function program block      DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 625 and 626 -- Error evaluating instruction: SPARK°mapLeftIndex°bestPipeline·FRAME·UNKNOWN·false°_fVar10444511·FRAME·UNKNOWN·false°1·SCALAR·INT64·true°1·SCALAR·INT64·true°1·SCALAR·INT64·true°4·SCALAR·INT64·true°_fVar10444512·FRAME·FP64°RIGHT      DMLRuntimeException -- FrameIndexingSPInstruction: The updated output dimensions are invalid: [-1 x 4, nnz=1000 (false), blocks (1000 x 1000)][0morg.apache.sysds.runtime.DMLRuntimeException: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:155) at org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:88) at org.apache.sysds.api.DMLScript.execute(DMLScript.java:423) at org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:266) at org.apache.sysds.api.DMLScript.main(DMLScript.java:164) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164) at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:149) ... 16 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 147 and 149 -- Error evaluating instruction: CP°fcall°.defaultNS°m_bandit°true°12°0°X_train=eX·MATRIX·FP64°Y_train=eY·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°targetList=targetClassification·LIST·UNKNOWN·false°lp=_fVar5386404·FRAME·UNKNOWN·false°primitives=primitives·FRAME·FP64·false°param=param·FRAME·FP64·false°k=5·SCALAR·INT64·true°R=30·SCALAR·INT64·true°verbose=true·SCALAR·BOOLEAN·true°output=file:/home/ssiddiqi/output/parfor//·SCALAR·STRING·true°index=1·SCALAR·INT64·true at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:199) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143) ... 17 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .defaultNS::m_bandit at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:198) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:250) ... 20 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 208 and 314 -- Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:142) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:191) ... 21 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:135) ... 22 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 282 and 285 -- Error evaluating instruction: CP°fcall°.defaultNS°extractBracketWinners°true°4°2°pipeline=_sbcvar12·MATRIX·FP64°hyperparam=_sbcvar13·MATRIX·FP64°k=5·SCALAR·INT64·true°conf=lookup·FRAME·FP64·false°bracket_bestPipeline°bracket_bestHyperparams at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:199) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143) ... 23 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .defaultNS::extractBracketWinners at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:198) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:250) ... 26 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 609 and 630 -- Error evaluating function program block at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:142) at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:191) ... 27 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164) at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:135) ... 28 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 625 and 626 -- Error evaluating instruction: SPARK°mapLeftIndex°bestPipeline·FRAME·UNKNOWN·false°_fVar10444511·FRAME·UNKNOWN·false°1·SCALAR·INT64·true°1·SCALAR·INT64·true°1·SCALAR·INT64·true°4·SCALAR·INT64·true°_fVar10444512·FRAME·FP64°RIGHT at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:199) at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125) at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143) ... 29 moreCaused by: org.apache.sysds.runtime.DMLRuntimeException: FrameIndexingSPInstruction: The updated output dimensions are invalid: [-1 x 4, nnz=1000 (false), blocks (1000 x 1000)] at org.apache.sysds.runtime.instructions.spark.FrameIndexingSPInstruction.checkValidOutputDimensions(FrameIndexingSPInstruction.java:169) at org.apache.sysds.runtime.instructions.spark.FrameIndexingSPInstruction.processInstruction(FrameIndexingSPInstruction.java:118) at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:250) ... 32 more

Attachments

Activity

People

Assignee:: Matthias Boehm

Reporter:: Shafaq Siddiqi

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 27/Apr/21 16:53

Updated:: 13/May/21 19:19

Resolved:: 13/May/21 19:19