[SYSTEMDS-2953] Parfor failure in Spark context

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemDS 2.1
• Assignee: Matthias Boehm
• Reporter: Shafaq Siddiqi
• Created: 27/Apr/21 16:51
• Updated: 06/Jun/21 19:26
• Resolved: 06/Jun/21 19:26

Description

While running the cleaning pipelines in Spark context the script terminated with following exception,

 

An Error Occured :
 DMLRuntimeException -- org.apache.sysds.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in parallel.
 DMLRuntimeException -- PARFOR: Failed to execute loop in parallel.
 DMLRuntimeException -- Not supported: Instructions of type other than CP instructions org.apache.sysds.runtime.instructions.spark.MultiReturnParameterizedBuiltinSPInstruction
SPARK°transformencode°F·FRAME·STRING·false°_Var5410935·SCALAR·STRING·false°X°M
org.apache.sysds.runtime.DMLRuntimeException: org.apache.sysds.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in parallel.
 at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:155)
 at org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:88)
 at org.apache.sysds.api.DMLScript.execute(DMLScript.java:423)
 at org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:266)
 at org.apache.sysds.api.DMLScript.main(DMLScript.java:164)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
 at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
 at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
 at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
 at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
 at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysds.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in parallel.
 at org.apache.sysds.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:649)
 at org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:149)
 ... 16 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: Not supported: Instructions of type other than CP instructions org.apache.sysds.runtime.instructions.spark.MultiReturnParameterizedBuiltinSPInstruction
SPARK°transformencode°F·FRAME·STRING·false°_Var5410935·SCALAR·STRING·false°X°M
 at org.apache.sysds.runtime.util.ProgramConverter.serializeInstructions(ProgramConverter.java:995)
 at org.apache.sysds.runtime.util.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1229)
 at org.apache.sysds.runtime.util.ProgramConverter.rSerializeProgramBlocks(ProgramConverter.java:1118)
 at org.apache.sysds.runtime.util.ProgramConverter.rSerializeProgramBlock(ProgramConverter.java:1222)
 at org.apache.sysds.runtime.util.ProgramConverter.rSerializeFunctionProgramBlocks(ProgramConverter.java:1104)
 at org.apache.sysds.runtime.util.ProgramConverter.serializeProgram(ProgramConverter.java:858)
 at org.apache.sysds.runtime.util.ProgramConverter.serializeParForBody(ProgramConverter.java:820)
 at org.apache.sysds.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:853)
 at org.apache.sysds.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:637)
 ... 17 more