Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7068

Fix Reduce Exception was overwrited by ReduceTask

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.1
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:
      None
    • Environment:

      CentOS 7

      Hadoop-2.7.1

      Hive-1.2.1

      Description

       

          try {
            //increment processed counter only if skipping feature is enabled
            boolean incrProcCount = SkipBadRecords.getReducerMaxSkipGroups(job)>0 &&
              SkipBadRecords.getAutoIncrReducerProcCount(job);
            
            ReduceValuesIterator<INKEY,INVALUE> values = isSkipping() ? 
                new SkippingReduceValuesIterator<INKEY,INVALUE>(rIter, 
                    comparator, keyClass, valueClass, 
                    job, reporter, umbilical) :
                new ReduceValuesIterator<INKEY,INVALUE>(rIter, 
                comparator, keyClass, valueClass,
                job, reporter);
            values.informReduceProgress();
            while (values.more()) {
              reduceInputKeyCounter.increment(1);
              reducer.reduce(values.getKey(), values, collector, reporter);
              if(incrProcCount) {
                reporter.incrCounter(SkipBadRecords.COUNTER_GROUP, 
                    SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS, 1);
              }
              values.nextKey();
              values.informReduceProgress();
            }
      
      
            reducer.close();
            reducer = null;
            
            out.close(reporter);
            out = null;
          } finally {
            IOUtils.cleanupWithLogger(LOG, reducer);
            closeQuietly(out, reporter);
          }
        }
      

      if  reducer.close(); throw Exception , reducer = null; will not run, then IOUtils.cleanupWithLogger(LOG, reducer); 

       

      will throw Exception and overwrite the Exception of reducer.close();

      so we should catch it and print log to help targeting issues

       

       

       

        Attachments

        1. MAPREDUCE_7068.patch
          0.9 kB
          tartarus

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tartarus tartarus
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: