[SYSTEMDS-3416] Cleaning Pipelines failed with No space left on device

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: SystemDS 3.1
• Fix Version/s: SystemDS 3.1
• Assignee: Unassigned
• Reporter: Shafaq Siddiqi
• Created: 04/Aug/22 09:18
• Updated: 15/Mar/23 19:13
• Resolved: 06/Aug/22 13:54

Description

Please refer the stake trace and the scripts to reproduce the error

22/08/04 01:40:48 ERROR UtilFunctions: Failed to close IO resource.
java.io.IOException: No space left on device
    at java.base/java.io.FileOutputStream.writeBytes(Native Method)
    at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
    at org.apache.sysds.runtime.util.FastBufferedDataOutput

An Error Occurred : 
      DMLRuntimeException -- org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 2 and 32 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar2850·FRAME·FP64·false°Xtrain=_mVar2853·MATRIX·FP64°Ytrain=_mVar2854·MATRIX·FP64°Xtest=_mVar2856·MATRIX·FP64°Ytest=_mVar2857·MATRIX·FP64°metaList=_Var2864·LIST·UNKNOWN·false°hyperParameters=_mVar2866·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°hpForPruning=_mVar2867·MATRIX·FP64°changesByOp=_mVar2867·MATRIX·FP64°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 2 and 32 -- Error evaluating instruction: CP°fcall°.builtinNS°s_executePipeline°true°12°8°pipeline=_fVar2850·FRAME·FP64·false°Xtrain=_mVar2853·MATRIX·FP64°Ytrain=_mVar2854·MATRIX·FP64°Xtest=_mVar2856·MATRIX·FP64°Ytest=_mVar2857·MATRIX·FP64°metaList=_Var2864·LIST·UNKNOWN·false°hyperParameters=_mVar2866·MATRIX·FP64°flagsCount=5·SCALAR·INT64·true°test=true·SCALAR·BOOLEAN·true°verbose=false·SCALAR·BOOLEAN·true°hpForPruning=_mVar2867·MATRIX·FP64°changesByOp=_mVar2867·MATRIX·FP64°trainX°trainy°testX°testy°Tr°hpForPruning°changesByOp°changesByPip
      DMLRuntimeException -- error executing function .builtinNS::s_executePipeline
      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 52 and 119 -- Error evaluating function program block
      DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
      DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 79 and 104 -- Error evaluating if statement body 
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 80 and 80 -- Error evaluating instruction: CP°eval°imputeByMean·SCALAR·STRING·true°hp·LIST·UNKNOWN·false°_Var3012·LIST·UNKNOWN
      DMLRuntimeException -- error executing function .builtinNS::m_imputeByMean
      DMLRuntimeException -- ERROR: Runtime error in function program block generated from function statement block between lines 36 and 64 -- Error evaluating function program block
      DMLRuntimeException -- ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
      DMLRuntimeException -- ERROR: Runtime error in if program block generated from if statement block between lines 44 and 48 -- Error evaluating if statement body 
      DMLRuntimeException -- ERROR: Runtime error in program block generated from statement block between lines 46 and 47 -- Error evaluating instruction: CP°rmempty°margin=rows°select=_mVar3121°empty.return=true°target=_mVar3119°bRmEmptyBC=false°_mVar3122·MATRIX·FP64
      DMLRuntimeException -- Eviction to local path /home/ssiddiqi/tmp//_p4099023_129.27.206.4//cache/cache000001105.dat (1542262772) failed.
              IOException -- No space left on device