Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6459

Native task crashes when merging spilled file on ppc64

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Environment:

      Linux version 2.6.32-431.el6.ppc64

    • Hadoop Flags:
      Reviewed

      Description

      when running native task on ppc64´╝îmerging spilled files fails since we could not deserialize local spill file correctly.
      Function readVLong in WritableUtils.h and Buffers.h, we try to compare a char with a number and convert a char to int64_t. It does not work correctly on ppc64 since char definition is different between ppc64 and x86 platform. On x86 platform char is defined as signed number while on ppc64 char is unsigned. As a result, we write EOF marker [-1, -1] at the end of spill partition, but deserialize chars as [255, 255].

        Attachments

        1. MAPREDUCE-6459.002.patch
          0.8 kB
          Tao Jie
        2. MAPREDUCE-6459.001.patch
          3 kB
          Tao Jie
        3. ppc64_error.txt
          12 kB
          Tao Jie

          Issue Links

            Activity

              People

              • Assignee:
                Tao Jie Tao Jie
                Reporter:
                Tao Jie Tao Jie
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: