Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
-
Component/s: mrv2, nodemanager
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
getMapOutputInfo expected the base parameter is getBaseLocation(jobId, user) + mapId
When it is called inside populateHeaders, the base parameter is set correctly
String base = outputBaseStr + mapId;
MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce, user);
When it is called outside populateHeaders, the base parameter is set wrongly to outputBasePathStr after number of mapId cached exceeds mapOutputMetaInfoCacheSize.
String outputBasePathStr = getBaseLocation(jobId, user);
MapOutputInfo info = mapOutputInfoMap.get(mapId);
if (info == null) {
info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
}
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Thanks for the review Devaraj K! Yes, that is a good suggestion. I uploaded a new patch MAPREDUCE-6425.001.patch which added a functional test. It fails without the code changes. Please review it.
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 15m 14s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 34s | There were no new javac warning messages. |
| +1 | javadoc | 9m 37s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 19s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 35s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 0m 44s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | mapreduce tests | 0m 19s | Tests passed in hadoop-mapreduce-client-shuffle. |
| 36m 21s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12743566/MAPREDUCE-6425.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2eae130 |
| hadoop-mapreduce-client-shuffle test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5868/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5868/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5868/console |
This message was automatically generated.
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 15m 16s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 33s | There were no new javac warning messages. |
| +1 | javadoc | 9m 38s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 21s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 38s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 38s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 0m 43s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | mapreduce tests | 0m 20s | Tests passed in hadoop-mapreduce-client-shuffle. |
| 36m 32s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12743594/MAPREDUCE-6425.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2eae130 |
| hadoop-mapreduce-client-shuffle test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5869/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5869/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5869/console |
This message was automatically generated.
Thanks zhihai xu for the updated patch.
+1, latest patch looks good to me, will commit it shortly.
FAILURE: Integrated in Hadoop-trunk-Commit #8114 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8114/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
- hadoop-mapreduce-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Yarn-trunk #978 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/978/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #248 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/248/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
- hadoop-mapreduce-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #246 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/246/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
- hadoop-mapreduce-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2194 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2194/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2175 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2175/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
- hadoop-mapreduce-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #236 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/236/)
MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to (devaraj: rev bff67dfe2f811654ffb1bbcbd87509c185f452b6)
- hadoop-mapreduce-project/CHANGES.txt
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
- hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
Nice catch zhihai xu, appreciate your efforts.
Can you add a functional test to demonstrate the issue and fails without the code changes?