Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2774

Add counters to show number of key/values that have been sorted and merged in the maps and reduces

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For each pass of the sort and merge, I would like a count of the number of records. So for example, if the map output 100 records and they were sorted once, the counter would be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level merge, it may not be a multiple of the number of map output records. This would let the users easily see if they have values like io.sort.mb or io.sort.factor set too low.

        Attachments

        1. HADOOP-2774.patch
          30 kB
          Ravi Gummadi
        2. HADOOP-2774.patch
          30 kB
          Ravi Gummadi
        3. HADOOP-2774.patch
          30 kB
          Ravi Gummadi
        4. HADOOP-2774.patch
          30 kB
          Ravi Gummadi
        5. HADOOP-2774.patch
          30 kB
          Ravi Gummadi
        6. HADOOP-2774.patch
          17 kB
          Ravi Gummadi
        7. HADOOP-2774.patch
          18 kB
          Ravi Gummadi

          Activity

            People

            • Assignee:
              ravidotg Ravi Gummadi
              Reporter:
              owen.omalley Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: