Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6063

In sortAndSpill of MapTask.java, size is calculated wrongly when bufend < bufstart.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: mrv1, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In sortAndSpill of MapTask.java, size is calculated wrongly when bufend < bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend.
      Should change

           long size = (bufend >= bufstart
                ? bufend - bufstart
                : (bufvoid - bufend) + bufstart) +
                        partitions * APPROX_HEADER_LENGTH;
      

      to:

           long size = (bufend >= bufstart
                ? bufend - bufstart
                : (bufvoid - bufstart) + bufend) +
                        partitions * APPROX_HEADER_LENGTH;
      

      It is because when wraparound happen (bufend < bufstart) , the size should
      bufvoid - bufstart (bigger one) + bufend(small one).
      You can find similar code implementation in MapTask.java:

              mapOutputByteCounter.increment(valend >= keystart
                  ? valend - keystart
                  : (bufvoid - keystart) + valend);
      

        Attachments

        1. MAPREDUCE-6063.000.patch
          1 kB
          Zhihai Xu
        2. MAPREDUCE-6063.branch-1.patch
          0.6 kB
          Zhihai Xu

          Activity

            People

            • Assignee:
              zxu Zhihai Xu
              Reporter:
              zxu Zhihai Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: