Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5649

Reduce cannot use more than 2G memory for the final merge

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method:

      int maxInMemReduce = (int)Math.min(
      Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE);

      This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts.

        Attachments

        1. MAPREDUCE-5649.001.patch
          6 kB
          Gera Shegalov
        2. MAPREDUCE-5649.002.patch
          6 kB
          Gera Shegalov
        3. MAPREDUCE-5649.003.patch
          6 kB
          Gera Shegalov

        Issue Links

          Activity

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              stanley_shi stanley shi

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment