Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5649

Reduce cannot use more than 2G memory for the final merge

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method:

      int maxInMemReduce = (int)Math.min(
      Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE);

      This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts.

      Attachments

        1. MAPREDUCE-5649.001.patch
          6 kB
          Gera Shegalov
        2. MAPREDUCE-5649.002.patch
          6 kB
          Gera Shegalov
        3. MAPREDUCE-5649.003.patch
          6 kB
          Gera Shegalov

        Issue Links

          Activity

            People

              jira.shegalov Gera Shegalov
              stanley_shi stanley shi
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: