Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6425

ShuffleHandler passes wrong "base" parameter to getMapOutputInfo if mapId is not in the cache.

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
      getMapOutputInfo expected the base parameter is getBaseLocation(jobId, user) + mapId
      When it is called inside populateHeaders, the base parameter is set correctly

              String base = outputBaseStr + mapId;
              MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce, user);
      

      When it is called outside populateHeaders, the base parameter is set wrongly to outputBasePathStr after number of mapId cached exceeds mapOutputMetaInfoCacheSize.

               String outputBasePathStr = getBaseLocation(jobId, user);
                MapOutputInfo info = mapOutputInfoMap.get(mapId);
                if (info == null) {
                  info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
                }
      

        Attachments

        1. MAPREDUCE-6425.001.patch
          6 kB
          zhihai xu
        2. MAPREDUCE-6425.000.patch
          2 kB
          zhihai xu

          Activity

            People

            • Assignee:
              zxu zhihai xu
              Reporter:
              zxu zhihai xu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: