Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7336

MapTask leaks sort buffer into merge increasing GC pressure in merge

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.2.2
    • None
    • performance
    • None

    Description

      When running with a large value for io.sort.mb, we see lots of GC pressure (and often GC failures) during the merge phase as the array of spill records are allocated. This is because the kvbuffer allocated during the sort phase is still allocated and consuming memory.

      This doesn't seem to be the intention; line 1539 of MapTask.java reads:

      // release sort buffer before the merge
      kvbuffer = null;

      But this isn't the only place that has a reference to kvbuffer; line 1002 wraps it into kvmeta:

      kvmeta = ByteBUffer.wrap(kvbuffer)

      The code needs to deference kvmeta so that kvbuffer can be garbage collected as desired.

      Attachments

        Activity

          People

            Unassigned Unassigned
            scott.oaks@oracle.com Scott Oaks
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: