Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2780

MapReduceLauncher should break early when one of the jobs throws an exception

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:

        jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
        ...
        jcThread.start();
        // Now wait, till we are finished.
        while(!jc.allFinished()){
        ...
        }
        //check for the jobControlException first
        //if the job controller fails before launching the jobs then there are
        //no jobs to check for failure
        if (jobControlException != null) {
          ...
        }
      

      There are two problems with this approach:
      1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
      2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.

        Attachments

        1. PIG-2780.1.patch
          6 kB
          Jie Li
        2. PIG-2780.0.patch
          5 kB
          Jie Li

          Activity

            People

            • Assignee:
              jay23jack Jie Li
              Reporter:
              pengfeng Feng Peng
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: