Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5649

Reduce cannot use more than 2G memory for the final merge

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method:

      int maxInMemReduce = (int)Math.min(
      Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE);

      This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts.

        Attachments

        1. MAPREDUCE-5649.003.patch
          6 kB
          Gera Shegalov
        2. MAPREDUCE-5649.002.patch
          6 kB
          Gera Shegalov
        3. MAPREDUCE-5649.001.patch
          6 kB
          Gera Shegalov

          Issue Links

            Activity

              People

              • Assignee:
                jira.shegalov Gera Shegalov
                Reporter:
                stanley_shi stanley shi
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: