Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4229

Counter names' memory usage can be decreased by interning

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.2, 2.0.2-alpha, 3.0.0-alpha1
    • 2.0.3-alpha, 0.23.5
    • jobtracker
    • None
    • Reviewed

    Description

      In our experience, most of the memory in production JTs goes to storing counter names (String objects and character arrays). Since most counter names are reused again and again, it would be a big memory savings to keep a hash set of already-used counter names within a job, and refer to the same object from all tasks.

      Attachments

        1. MAPREDUCE-4229.patch
          17 kB
          Miomir Boljanovic
        2. MAPREDUCE-4229-branch-0.23.patch
          16 kB
          Miomir Boljanovic
        3. MAPREDUCE-4229-branch-0.23.patch
          8 kB
          Miomir Boljanovic
        4. mr-4229.txt
          16 kB
          Robert Joseph Evans
        5. MR-4229.txt
          16 kB
          Robert Joseph Evans

        Issue Links

          Activity

            People

              miomir Miomir Boljanovic
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: