Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: None
    • Component/s: mrv2, task
    • Labels:
      None

      Description

      My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage

        Issue Links

          Activity

          Ravi Prakash made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Target Version/s 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320060, 12320354, 12320355 ] 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320354, 12320060, 12320355 ]
          Resolution Duplicate [ 3 ]
          Ravi Prakash made changes -
          Link This issue is duplicated by MAPREDUCE-2264 [ MAPREDUCE-2264 ]
          Hide
          Ravi Prakash added a comment -

          This seems to be a dup of MAPREDUCE-2264. Closing as such.

          Show
          Ravi Prakash added a comment - This seems to be a dup of MAPREDUCE-2264 . Closing as such.
          Andrew Hancock made changes -
          Affects Version/s 0.22.0 [ 12314184 ]
          Target Version/s 3.0.0, 2.0.0, 0.23.3 [ 12320355, 12320354, 12320060 ] 0.23.3, 2.0.0, 3.0.0 [ 12320060, 12320354, 12320355 ]
          Hide
          Andrew Hancock added a comment -

          I am experiencing this bug in version 0.22 as well.

          It happens any time I turn on the following config in my job:

          config.setFloat("mapreduce.reduce.input.buffer.percent", 0.1f);

          If I remove this setting, the bug goes away.

          When this option is turned on, the following symptoms are exhibited:
          1. During the reduce phase many of the reduce tasks will quickly read > 100%. The Job will then report 100% complete.

          2. I have on occasion seen it then go back below 100% on the overall progress. It will oscillate between different numbers in that case.

          3. The job appears to complete normally in spite of the progress reporting issues.

          Show
          Andrew Hancock added a comment - I am experiencing this bug in version 0.22 as well. It happens any time I turn on the following config in my job: config.setFloat("mapreduce.reduce.input.buffer.percent", 0.1f); If I remove this setting, the bug goes away. When this option is turned on, the following symptoms are exhibited: 1. During the reduce phase many of the reduce tasks will quickly read > 100%. The Job will then report 100% complete. 2. I have on occasion seen it then go back below 100% on the overall progress. It will oscillate between different numbers in that case. 3. The job appears to complete normally in spite of the progress reporting issues.
          Robert Joseph Evans made changes -
          Target Version/s 0.23.1 [ 12318883 ] 0.23.3, 2.0.0, 3.0.0 [ 12320060, 12320354, 12320355 ]
          Hide
          Todd Lipcon added a comment -

          This was on 0.23.0 before release, so might be fixed by now. I do think I was using intermediate compression and maybe some of the "in-memory merge" config options. I think this was the config I used for the job:

            <property>
              <name>io.sort.mb</name>
              <value>650</value>
            </property>
            <property>
              <name>mapreduce.map.sort.spill.percent</name>
              <value>0.98</value>
            </property>
            <property>
              <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
              <value>0.8</value>
            </property>
            <property>
              <name>mapreduce.reduce.input.buffer.percent</name>
              <value>0.8</value>
            </property>
          
            <property>
              <name>io.sort.factor</name>
              <value>100</value>
            </property>
          

          and it was a terasort

          Show
          Todd Lipcon added a comment - This was on 0.23.0 before release, so might be fixed by now. I do think I was using intermediate compression and maybe some of the "in-memory merge" config options. I think this was the config I used for the job: <property> <name>io.sort.mb</name> <value>650</value> </property> <property> <name>mapreduce.map.sort.spill.percent</name> <value>0.98</value> </property> <property> <name>mapreduce.reduce.shuffle.input.buffer.percent</name> <value>0.8</value> </property> <property> <name>mapreduce.reduce.input.buffer.percent</name> <value>0.8</value> </property> <property> <name>io.sort.factor</name> <value>100</value> </property> and it was a terasort
          Ravi Prakash made changes -
          Assignee Ravi Prakash [ raviprak ]
          Hide
          Ravi Prakash added a comment -

          Hi Todd!
          Any clues on how this may be reproduced?

          Show
          Ravi Prakash added a comment - Hi Todd! Any clues on how this may be reproduced?
          Arun C Murthy made changes -
          Priority Critical [ 2 ] Major [ 3 ]
          Arun C Murthy made changes -
          Field Original Value New Value
          Target Version/s 0.23.0 [ 12315570 ] 0.23.1 [ 12318883 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development