[PIG-2780] MapReduceLauncher should break early when one of the jobs throws an exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11
• Hadoop Flags: Reviewed
• Assignee: Jie Li
• Reporter: Feng Peng
• Created: 29/Jun/12 18:48
• Updated: 22/Feb/13 04:53
• Resolved: 12/Jul/12 07:36

Description

Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:

  jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
  ...
  jcThread.start();
  // Now wait, till we are finished.
  while(!jc.allFinished()){
  ...
  }
  //check for the jobControlException first
  //if the job controller fails before launching the jobs then there are
  //no jobs to check for failure
  if (jobControlException != null) {
    ...
  }

There are two problems with this approach:
1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.