Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6724

Single shuffle to memory must not exceed Integer#MAX_VALUE

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When shuffle is done in memory, MergeManagerImpl converts the requested size to an int to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.

        Attachments

        1. mapreduce6724.001.patch
          8 kB
          Haibo Chen
        2. mapreduce6724.002.patch
          8 kB
          Haibo Chen
        3. mapreduce6724.003.patch
          7 kB
          Haibo Chen
        4. mapreduce6724.004.patch
          7 kB
          Haibo Chen
        5. mapreduce6724.005.patch
          7 kB
          Haibo Chen
        6. mapreduce6724.006.patch
          8 kB
          Haibo Chen
        7. mapreduce6724.007.patch
          4 kB
          Haibo Chen
        8. mapreduce6724.008.patch
          4 kB
          Haibo Chen
        9. MAPREDUCE-6724.009.patch
          5 kB
          Gera Shegalov

          Issue Links

            Activity

              People

              • Assignee:
                haibochen Haibo Chen
                Reporter:
                haibochen Haibo Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: