[SYSTEMDS-3127] Cleaning pipeline enumeration w/ dimension mismatches - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: SystemDS 2.2
Component/s: None
Labels:
None

Description

The call to abstain through parfor-eval causes race conditions on recompilation because all threads access the same multiLogReg function. This requires a fix of the parfor-eval function preparation, but also the abstain builtin which should not call multilogreg directly but use a user-defined function for classification/regression. cc ssiddiqi

org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°./scripts/builtin/bandit.dml°run_with_hyperparam_t1154°true°15°3°lp=_fVar2787984·FRAME·UNKNOWN·false°ph_pip=_fVar2787990·FRAME·FP64·false°r_i=2·SCALAR·INT64·true°X=X·MATRIX·FP64°Y=y·MATRIX·FP64°Xtest=Xtest·MATRIX·FP64°Ytest=ytest·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°evaluationFunc=evalML·SCALAR·STRING·true°evalFunHp=evalFunHp·MATRIX·FP64°param=param·FRAME·UNKNOWN·false°featureFrameOuter=_fVar2787992·FRAME·STRING·false°cv=false·SCALAR·BOOLEAN·true°cvk=3·SCALAR·INT64·true°verbose=false·SCALAR·BOOLEAN·true°outPip°outHp°feaFrameOuter
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:289)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:203)
	at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
	at org.apache.sysds.runtime.controlprogram.parfor.ParWorker.executeSetTask(ParWorker.java:154)
	at org.apache.sysds.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:121)
	at org.apache.sysds.runtime.controlprogram.parfor.LocalParWorker.run(LocalParWorker.java:121)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function ./scripts/builtin/bandit.dml::run_with_hyperparam_t1154
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:254)
	... 6 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:143)
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201)
	... 7 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
	at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164)
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:136)
	... 8 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
	at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164)
	at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143)
	... 9 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body 
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113)
	at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143)
	... 10 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline_t1154°true°13°5°logical=lp·FRAME·UNKNOWN·false°pipeline=ph_pip·FRAME·UNKNOWN·false°X=X·MATRIX·FP64°Y=Y·MATRIX·FP64°Xtest=Xtest·MATRIX·FP64°Ytest=Ytest·MATRIX·FP64°metaList=metaList·LIST·UNKNOWN·false°hyperParameters=hp_matrix·MATRIX·FP64°hpForPruning=hpForPruning·MATRIX·FP64°changesByOp=changesByOp·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°eXtrain°eYtrain°eXtest°eYtest°Tr
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:289)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:203)
	at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106)
	... 11 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::s_executePipeline_t1154
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:254)
	... 14 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:143)
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201)
	... 15 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
	at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:164)
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:136)
	... 16 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating else statement body 
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:126)
	at org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:143)
	... 17 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°eval°abstain·SCALAR·STRING·true°hp·LIST·UNKNOWN·false°_mVar3281765·MATRIX·FP64
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:289)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:203)
	at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:119)
	... 18 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::m_abstain_t1154
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208)
	at org.apache.sysds.runtime.instructions.cp.EvalNaryCPInstruction.processInstruction(EvalNaryCPInstruction.java:148)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:254)
	... 21 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 0 and 0 -- Error evaluating function program block
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:143)
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201)
	... 23 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 0 and 0 -- Error evaluating if statement body 
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113)
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:136)
	... 24 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°fcall°.builtinNS°m_multiLogReg°false°8°1°X=X·MATRIX·FP64°Y=Y·MATRIX·FP64°icpt=1·SCALAR·INT64·true°reg=1.0E-4·SCALAR·FP64·true°maxi=100·SCALAR·INT64·true°maxii=0·SCALAR·INT64·true°verbose=false·SCALAR·BOOLEAN·true°tol=1.0E-6·SCALAR·FP64·true°betas
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:289)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:203)
	at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106)
	... 25 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: error executing function .builtinNS::m_multiLogReg
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:208)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:254)
	... 28 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 49 and 283 -- Error evaluating function program block
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:143)
	at org.apache.sysds.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:201)
	... 29 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in if program block generated from if statement block between lines 72 and 75 -- Error evaluating if statement body 
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:113)
	at org.apache.sysds.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:136)
	... 30 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 73 and 74 -- Error evaluating instruction: CP°append°X·MATRIX·FP64°_mVar3284144·MATRIX·FP64°152·SCALAR·INT64·true°_mVar3284145·MATRIX·FP64°true
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:289)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:203)
	at org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
	at org.apache.sysds.runtime.controlprogram.IfProgramBlock.execute(IfProgramBlock.java:106)
	... 31 more
Caused by: org.apache.sysds.runtime.DMLRuntimeException: Append-cbind is not possible for input matrices X and _mVar3284144 with different number of rows: 385 vs 338
	at org.apache.sysds.runtime.instructions.cp.MatrixAppendCPInstruction.validateInput(MatrixAppendCPInstruction.java:59)
	at org.apache.sysds.runtime.instructions.cp.MatrixAppendCPInstruction.processInstruction(MatrixAppendCPInstruction.java:45)
	at org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:254)
	... 34 more

Attachments

Activity

People

Assignee:: Matthias Boehm

Reporter:: Matthias Boehm

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 10/Sep/21 13:16

Updated:: 10/Sep/21 14:34

Resolved:: 10/Sep/21 14:34