Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Duplicate
-
Affects Version/s: 0.22.0, 0.23.0
-
Fix Version/s: None
-
Labels:None
-
Target Version/s:
Description
My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
Issue Links
- is duplicated by
-
MAPREDUCE-2264
Job status exceeds 100% in some cases
-
- Closed
-
Activity
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Target Version/s | 0.23.3, 2.0.0-alpha, 3.0.0 [ 12320060, 12320354, 12320355 ] | 2.0.0-alpha, 0.23.3, 3.0.0 [ 12320354, 12320060, 12320355 ] |
| Resolution | Duplicate [ 3 ] |
| Link |
This issue is duplicated by |
| Affects Version/s | 0.22.0 [ 12314184 ] | |
| Target Version/s | 3.0.0, 2.0.0, 0.23.3 [ 12320355, 12320354, 12320060 ] | 0.23.3, 2.0.0, 3.0.0 [ 12320060, 12320354, 12320355 ] |
I am experiencing this bug in version 0.22 as well.
It happens any time I turn on the following config in my job:
config.setFloat("mapreduce.reduce.input.buffer.percent", 0.1f);
If I remove this setting, the bug goes away.
When this option is turned on, the following symptoms are exhibited:
1. During the reduce phase many of the reduce tasks will quickly read > 100%. The Job will then report 100% complete.
2. I have on occasion seen it then go back below 100% on the overall progress. It will oscillate between different numbers in that case.
3. The job appears to complete normally in spite of the progress reporting issues.
| Target Version/s | 0.23.1 [ 12318883 ] | 0.23.3, 2.0.0, 3.0.0 [ 12320060, 12320354, 12320355 ] |
This was on 0.23.0 before release, so might be fixed by now. I do think I was using intermediate compression and maybe some of the "in-memory merge" config options. I think this was the config I used for the job:
<property>
<name>io.sort.mb</name>
<value>650</value>
</property>
<property>
<name>mapreduce.map.sort.spill.percent</name>
<value>0.98</value>
</property>
<property>
<name>mapreduce.reduce.shuffle.input.buffer.percent</name>
<value>0.8</value>
</property>
<property>
<name>mapreduce.reduce.input.buffer.percent</name>
<value>0.8</value>
</property>
<property>
<name>io.sort.factor</name>
<value>100</value>
</property>
and it was a terasort
| Assignee | Ravi Prakash [ raviprak ] |
Hi Todd!
Any clues on how this may be reproduced?
| Priority | Critical [ 2 ] | Major [ 3 ] |
| Field | Original Value | New Value |
|---|---|---|
| Target Version/s | 0.23.0 [ 12315570 ] | 0.23.1 [ 12318883 ] |
This seems to be a dup of
MAPREDUCE-2264. Closing as such.