[UIMA-5883] DUCC JobDriver (JD) may cause job to never process all work items if JobProcess (JP) is preempted

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-Ducc
• Component/s: DUCC
• Assignee: Lou DeGenaro
• Reporter: Lou DeGenaro
• Created: 04/Oct/18 20:25
• Updated: 05/Oct/18 11:37
• Resolved: 05/Oct/18 11:37

Description

Noticed on Apache DUCC demo that Job 14493 had work items total=10001, completed=9986, dispatch=15, and made no further progress.  Looking in work-item-state.json we see the 9986 that have completed and can infer precisely those that did not.  Then looking in the JD log for those not yet complete work items, we see entries similar to:

25 Sep 2018 23:28:28,042 WARN ActionGet - T[14] engage seqNo=? remote=uima-ducc-demo-6.8448.25 node=uima-ducc-demo-6 pid=8448 text=process discontinued

Looking at the code, we see that under this condition that the JD has obtained a CAS from the CR, but chooses not to give it to the requesting JP process since JD knows that the requester has been targeted for termination (e.g. preempted).  But the JD forgets to put the CAS back into the queue!  And therefore those CASes never get processed and the Job is hung forevermore.