Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6724

Single shuffle to memory must not exceed Integer#MAX_VALUE

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 2.7.4, 3.0.0-alpha1
    • mrv2
    • None
    • Reviewed

    Description

      When shuffle is done in memory, MergeManagerImpl converts the requested size to an int to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.

      Attachments

        1. mapreduce6724.001.patch
          8 kB
          Haibo Chen
        2. mapreduce6724.002.patch
          8 kB
          Haibo Chen
        3. mapreduce6724.003.patch
          7 kB
          Haibo Chen
        4. mapreduce6724.004.patch
          7 kB
          Haibo Chen
        5. mapreduce6724.005.patch
          7 kB
          Haibo Chen
        6. mapreduce6724.006.patch
          8 kB
          Haibo Chen
        7. mapreduce6724.007.patch
          4 kB
          Haibo Chen
        8. mapreduce6724.008.patch
          4 kB
          Haibo Chen
        9. MAPREDUCE-6724.009.patch
          5 kB
          Gera Shegalov

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6805 Negative Array Exception while reducer fetch mapper output

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              haibochen Haibo Chen
              haibochen Haibo Chen
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: