Pig
  1. Pig
  2. PIG-2780

MapReduceLauncher should break early when one of the jobs throws an exception

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:

        jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
        ...
        jcThread.start();
        // Now wait, till we are finished.
        while(!jc.allFinished()){
        ...
        }
        //check for the jobControlException first
        //if the job controller fails before launching the jobs then there are
        //no jobs to check for failure
        if (jobControlException != null) {
          ...
        }
      

      There are two problems with this approach:
      1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
      2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.

      1. PIG-2780.1.patch
        6 kB
        Jie Li
      2. PIG-2780.0.patch
        5 kB
        Jie Li

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Jie Li
            Reporter:
            Feng Peng
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development