Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5008

Merger progress miscounts with respect to EOF_MARKER

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After MAPREDUCE-2264, a segment's raw data length is calculated without the EOF_MARKER bytes. However, when the merge is counting how many bytes it processed, it includes the marker. This can cause the merge progress to go above 100%.

      Whether these EOF_MARKER bytes should count should be consistent between the two.

      This a JIRA instead of an amendment because MAPREDUCE-2264 already went into 2.0.3.

      1. MAPREDUCE-5008-branch-1.patch
        2 kB
        Sandy Ryza
      2. MAPREDUCE-5008.patch
        2 kB
        Sandy Ryza
      3. MAPREDUCE-5008.patch
        2 kB
        Sandy Ryza

        Activity

        Sandy Ryza created issue -
        Sandy Ryza made changes -
        Field Original Value New Value
        Description After MAPREDUCE-2264, a segment's raw data length is calculated without the EOF_MARKER bytes. However, when the merge is counting how many bytes it processed, it includes the marker. This can cause the merge progress to go above 100%.

        Whether these EOF_MARKER bytes should count should be consistent between the two.
        After MAPREDUCE-2264, a segment's raw data length is calculated without the EOF_MARKER bytes. However, when the merge is counting how many bytes it processed, it includes the marker. This can cause the merge progress to go above 100%.

        Whether these EOF_MARKER bytes should count should be consistent between the two.

        This a JIRA instead of an amendment because MAPREDUCE-2264 already went into 2.0.3.
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5008.patch [ 12569683 ]
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5008-branch-1.patch [ 12569684 ]
        Sandy Ryza made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Sandy Ryza made changes -
        Assignee Sandy Ryza [ sandyr ]
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5008.patch [ 12569897 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 1.2.0 [ 12321661 ]
        Fix Version/s 2.0.4-beta [ 12324032 ]
        Resolution Fixed [ 1 ]
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Sandy Ryza
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development