Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-18

Under load the shuffle sometimes gets incorrect data

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      This patch adds the mapid and reduceid in the http header of mapoutput when being sent to reduce node. Also validates compressed length, decompressed length, mapid and reduceid from http header at reduce node.

      Description

      While testing HADOOP-5223 under load, we found reduces receiving completely incorrect data. It was often random, but sometimes was the output of the wrong map for the wrong map. It appears to either be a Jetty or JVM bug, but it is clearly happening on the server side. In the HADOOP-5223 code, I added information about the map and reduce that were included and we should add similar protection to 0.20 and trunk.

      1. MR-18-0.20.patch
        6 kB
        Ravi Gummadi
      2. MR-18.v1.patch
        6 kB
        Ravi Gummadi
      3. MR-18.v1.1.patch
        6 kB
        Ravi Gummadi
      4. MR-18.patch
        5 kB
        Ravi Gummadi

        Activity

        Hide
        dhruba borthakur added a comment -

        which release of hadoop has this behaviour been seen?

        Show
        dhruba borthakur added a comment - which release of hadoop has this behaviour been seen?
        Hide
        Ravi Gummadi added a comment -

        The issue was verison 0.20. Owen, Please correct me if I am wrong.

        Show
        Ravi Gummadi added a comment - The issue was verison 0.20. Owen, Please correct me if I am wrong.
        Hide
        Owen O'Malley added a comment -

        It was trunk after 0.20, but based on my understanding of the likely cause, it would also affect 0.20.0

        Show
        Owen O'Malley added a comment - It was trunk after 0.20, but based on my understanding of the likely cause, it would also affect 0.20.0
        Hide
        Ravi Gummadi added a comment -

        Attaching patch for trunk.
        Please review and provide your comments.

        Show
        Ravi Gummadi added a comment - Attaching patch for trunk. Please review and provide your comments.
        Hide
        Jothi Padmanabhan added a comment -

        Some comments:

        1. On header verification failures, we should return null instead of continuing
        2. The validation for from_map_id of the header against the expected map id can be added in addition to validating if the map id is in the correct format
        Show
        Jothi Padmanabhan added a comment - Some comments: On header verification failures, we should return null instead of continuing The validation for from_map_id of the header against the expected map id can be added in addition to validating if the map id is in the correct format
        Hide
        Ravi Gummadi added a comment -

        Attaching patch with the suggested changes.

        Show
        Ravi Gummadi added a comment - Attaching patch with the suggested changes.
        Hide
        Jothi Padmanabhan added a comment -

        +1. Patch looks fine.

        Show
        Jothi Padmanabhan added a comment - +1. Patch looks fine.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch for branch 0.20

        Show
        Ravi Gummadi added a comment - Attaching patch for branch 0.20
        Hide
        Ravi Gummadi added a comment -

        Attaching new patch that applies to trunk. No extra code changes compared to earlier patch.

        Show
        Ravi Gummadi added a comment - Attaching new patch that applies to trunk. No extra code changes compared to earlier patch.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Ravi!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Ravi!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/25/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #25 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/25/ )
        Hide
        Ravi Gummadi added a comment -

        This doesn't need a unit test as the code changes are tested by many other testcases.

        Show
        Ravi Gummadi added a comment - This doesn't need a unit test as the code changes are tested by many other testcases.

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development