Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5968

Work directory is not deleted when downloadCacheObject throws IOException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1
    • Fix Version/s: 1.3.0
    • Component/s: mrv1
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Work directory is not deleted in DistCache if Exception happen in downloadCacheObject. In downloadCacheObject, the cache file will be copied to temporarily work directory first, then the work directory will be renamed to the final directory. If IOException happens during the copy, the work directory will not be deleted. This will cause garbage data left in local disk cache. For example If the MR application use Distributed Cache to send a very large Archive/file(50G), if the disk is full during the copy, then the IOException will be triggered, the work directory will be not deleted or renamed and the work directory will occupy a big chunk of disk space.

        Attachments

          Activity

            People

            • Assignee:
              zxu zhihai xu
              Reporter:
              zxu zhihai xu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: