Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4229

Counter names' memory usage can be decreased by interning

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.2, 2.0.2-alpha, 3.0.0-alpha1
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: jobtracker
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In our experience, most of the memory in production JTs goes to storing counter names (String objects and character arrays). Since most counter names are reused again and again, it would be a big memory savings to keep a hash set of already-used counter names within a job, and refer to the same object from all tasks.

        Attachments

        1. MR-4229.txt
          16 kB
          Robert Joseph Evans
        2. mr-4229.txt
          16 kB
          Robert Joseph Evans
        3. MAPREDUCE-4229-branch-0.23.patch
          8 kB
          Miomir Boljanovic
        4. MAPREDUCE-4229-branch-0.23.patch
          16 kB
          Miomir Boljanovic
        5. MAPREDUCE-4229.patch
          17 kB
          Miomir Boljanovic

          Issue Links

            Activity

              People

              • Assignee:
                miomir Miomir Boljanovic
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: