Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-90

incrementing counters should not be used for triggering record skipping

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The following code is really problematic:

      public void incrCounter(String group, String counter, long amount) {
        if (counters != null) {
          counters.incrCounter(group, counter, amount);
        }
        if(skipping && SkipBadRecords.COUNTER_GROUP.equals(group) && (
           SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS.equals(counter) ||
           SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS.equals(counter))) {
           //if application reports the processed records, move the 
           //currentRecStartIndex to the next.
           //currentRecStartIndex is the start index which has not yet been 
           //finished and is still in task's stomach.
           for(int i=0;i<amount;i++) {
              currentRecStartIndex = currentRecIndexIterator.next();
           }
         ...
      }
      

      In particular, if the user updates a counter with the wrong name, bad things will presumably happen...

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development