[MAPREDUCE-7068] Fix Reduce Exception was overwrited by ReduceTask

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.1
• Component/s: mrv1

Environment

CentOS 7

Hadoop-2.7.1

Hive-1.2.1

• Assignee: Unassigned
• Reporter: tartarus
• Created: 22/Mar/18 13:32
• Updated: 22/Mar/18 16:21

Description

 

    try {
      //increment processed counter only if skipping feature is enabled
      boolean incrProcCount = SkipBadRecords.getReducerMaxSkipGroups(job)>0 &&
        SkipBadRecords.getAutoIncrReducerProcCount(job);
      
      ReduceValuesIterator<INKEY,INVALUE> values = isSkipping() ? 
          new SkippingReduceValuesIterator<INKEY,INVALUE>(rIter, 
              comparator, keyClass, valueClass, 
              job, reporter, umbilical) :
          new ReduceValuesIterator<INKEY,INVALUE>(rIter, 
          comparator, keyClass, valueClass,
          job, reporter);
      values.informReduceProgress();
      while (values.more()) {
        reduceInputKeyCounter.increment(1);
        reducer.reduce(values.getKey(), values, collector, reporter);
        if(incrProcCount) {
          reporter.incrCounter(SkipBadRecords.COUNTER_GROUP, 
              SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS, 1);
        }
        values.nextKey();
        values.informReduceProgress();
      }


      reducer.close();
      reducer = null;
      
      out.close(reporter);
      out = null;
    } finally {
      IOUtils.cleanupWithLogger(LOG, reducer);
      closeQuietly(out, reporter);
    }
  }

if  reducer.close(); throw Exception , reducer = null; will not run, then IOUtils.cleanupWithLogger(LOG, reducer); 

 

will throw Exception and overwrite the Exception of reducer.close();

so we should catch it and print log to help targeting issues