-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
-
Component/s: mrv2, nodemanager
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
getMapOutputInfo expected the base parameter is getBaseLocation(jobId, user) + mapId
When it is called inside populateHeaders, the base parameter is set correctly
String base = outputBaseStr + mapId;
MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce, user);
When it is called outside populateHeaders, the base parameter is set wrongly to outputBasePathStr after number of mapId cached exceeds mapOutputMetaInfoCacheSize.
String outputBasePathStr = getBaseLocation(jobId, user); MapOutputInfo info = mapOutputInfoMap.get(mapId); if (info == null) { info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user); }