Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6105

Inconsistent configuration in property mapreduce.reduce.shuffle.merge.percent

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 2.4.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Similar to MAPREDUCE-5906, In MergeManagerImpl.java, the default value of MRJobConfig.SHUFFLE_MERGE_PERCENT(mapreduce.reduce.shuffle.merge.percent) should be 0.66
      According to official doc.
      https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

          this.mergeThreshold = (long)(this.memoryLimit * 
                                jobConf.getFloat(MRJobConfig.SHUFFLE_MERGE_PERCENT, 
                                                 0.90f));
      
      

        Attachments

        1. MAPREDUCE-6105.patch
          3 kB
          Harsh J
        2. MAPREDUCE-6105-01.patch
          1 kB
          Ray Chiang

          Activity

            People

            • Assignee:
              rchiang Ray Chiang
              Reporter:
              dongwook Dongwook Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: